—

Stream Processing and Analytics

Message Delivery Semantics

Published: 2026-08-07

Level: postgraduate

Audience: Postgraduate students in Stream Processing and Analytics

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

The rate mismatch problem (ingestion vs processing) - covered in Lecture 4
The layered streaming architecture (collection, data flow, and analysis tiers) - covered in Lecture 4
Message delivery semantics (introduction) - covered in Lecture 4
Horizontal scaling of the processing engine - covered in Lecture 4
Sampling-based processing, when a representative picture of the stream is enough - covered in Lecture 4

A streaming pipeline moves events from the place they are born (the collection tier) through a data flow layer into a processing layer (also called the analytics or analysis layer), and from there to downstream activities. This session answers a deceptively simple question that every such pipeline must face: when messages pile up or connections break, what are we allowed to lose, and what must we keep at any cost? The answer is codified in message delivery semantics — a small set of guarantees about how a message moves from producer to consumer.

The session builds the answer in four steps. First it shows why the question exists at all: when the rate of ingestion outruns the rate of processing, queues flood and messages die, so a pipeline must decide in advance how much loss it can live with. Second it names the three possible guarantees — at most once, at least once, and exactly once — and the three questions that choose between them for any real use case. Third it applies that method to three worked scenarios: transaction data monitoring, a recommendation system, and operational monitoring. Along the way the session settles the quiz and assignment schedule, and closes by checking where the course stands in its syllabus.

6.1 Motivation: When Ingestion Outruns Processing

The hook. Imagine a pipeline that receives 300 messages every minute while its processing engine can handle only 10. Where do the other 290 go each minute? This section shows that they pile up in a queue until the queue breaks — and that the only honest response is to decide, up front, how much of that loss your business can tolerate.

6.1.1 The Streaming Pipeline and the Two Rates

Every streaming architecture you design has the same backbone: data is collected at one end, carried through a data flow layer, processed or analyzed in the middle, and pushed toward downstream consumers. Earlier sessions covered the collection tier in detail, and the data flow layer's two interaction patterns — client-server and publisher-subscriber (PubSub). The questions in this session live in the data flow layer, where two speeds meet.

The first speed is the rate of ingestion, written $R_{I}$ — how fast data arrives at the pipeline. The second is the rate of processing, written $R_{P}$ — how fast the processing layer can actually work through messages. The design decision starts with comparing the two, by taking the ratio of ingestion to processing:

$\frac{R _{I}}{R _{P}}$

The ratio as a health check. $R_{I}$ (read "R sub I") is the number of messages arriving per unit of time. $R_{P}$ (read "R sub P") is the number of messages the processing layer can handle in the same unit of time. The units are "messages per unit time" — the exact unit was never pinned down in class, so think of them as comparable counts per second or minute.

The ratio $\frac{R _{I}}{R _{P}}$ is a pure number: the units of "messages per unit time" cancel on top and bottom. It answers one question directly: how many messages arrive while one message is being processed? A ratio of 1 means arrivals and processing are perfectly matched. A ratio of 2 means two messages arrive for every one processed. A ratio of 30 means thirty messages arrive for every one processed — and that is a pipeline in trouble.

If the two rates are comparable, there is no real problem. The pipeline can get by with an auxiliary queue: incoming messages sit in the queue, and the processing layer picks them up as it finishes each record. The queue absorbs the small, momentary mismatch between production and consumption.

Intuition — a single ticket counter. Think of the pipeline as one ticket window at a railway station. $R_{I}$ is the number of passengers joining the line per minute; $R_{P}$ is the number of tickets the clerk can issue per minute. While the line is short, the waiting area (the queue) smooths things out: passengers wait a few moments, the clerk works at their own pace, and everyone is served. The analogy holds while arrivals roughly match service speed. It breaks the moment arrivals dwarf service speed: the waiting area fills, passengers spill onto the platform, and some simply never get served — those are the lost messages.

6.1.2 Back Pressure

Real deployments are rarely that tidy. When ingestion outruns processing by a large margin, the pipeline enters a condition called back pressure. The session's concrete picture: when $R_{I} / R_{P}$ reaches the order of 30,

$\frac{R _{I}}{R _{P}} \approx 30$

then by the time the processing layer finishes one record, thirty records are already waiting. The queue floods. Messages pile up, the queue breaks, and messages are lost. So a ratio like 30:1 is a direct statement: this pipeline is losing data, whether you planned for it or not.

Worked example — a pipeline at 30:1.

Let the ingestion rate be $R_{I} = 300$ messages per minute and the processing rate be $R_{P} = 10$ messages per minute. Plug into the ratio:

$\frac{R _{I}}{R _{P}} = \frac{300}{10} = 30$

The processing layer takes $\frac{1}{R _{P}} = \frac{1}{10}$ minutes — that is 6 seconds — to finish one record. In those same 6 seconds the pipeline receives:

$R_{I} \times \frac{6}{60} = 300 \times 0.1 = 30 messages$

So while one message is being processed, 30 new ones arrive. Net backlog each minute: $300 - 10 = 290$ messages. A queue that can hold 30 messages floods in seconds; even a queue that can hold a few thousand is gone in under ten minutes. A ratio of 30:1 is not a warning that data may be lost — it is a statement that data is already being lost.

Sense-check: if processing were as fast as ingestion ( $R_{I} = R_{P}$ ), the ratio would be 1 and nothing would accumulate. Our 300-to-10 example makes the 30:1 figure come out of real arithmetic rather than thin air, and the 290-per-minute backlog grows in the right direction: faster arrivals than service means the line grows.

Visual intuition. Picture a bar chart with time on the horizontal axis (minutes) and number of messages on the vertical axis. One bar shows arrivals per minute, flat at 300; the other shows processing per minute, flat at 10. The arrivals bar towers over the processing bar, and the gap between the two — the white space above the short bar — is exactly the backlog: 290 messages every minute. Below the chart, imagine a second line: the queue's length climbing like a staircase, one 290-message step per minute, until it hits the queue's capacity ceiling and simply stops recording what cannot be stored. The takeaway: where the bars diverge, the queue grows; where the queue ends, the data ends with it.

Assumptions & scope. The auxiliary-queue fix assumes the mismatch between $R_{I}$ and $R_{P}$ is small and momentary — a burst that the queue can absorb and then drain. It also assumes the queue's storage is large enough to hold the backlog that actually forms. When the mismatch is a sustained 30:1, both assumptions fail: the queue neither drains nor fits, and messages spill. The back-pressure condition is the boundary case where the queue stops being a buffer and starts being a casualty.

Where does this leave a designer who is told to "just add a bigger queue"? The answer is nowhere: back pressure is a flow problem, not a storage problem. The mismatch compounds minute after minute, so a few minutes of 30:1 traffic is enough to exhaust any realistic queue. The fix has to come from the rates themselves — fewer messages accepted, or more messages processed.

Pitfalls.

Assuming the queue is infinite. A queue is a bounded buffer. The moment the backlog exceeds its capacity, messages are dropped or rejected. There is no "growing queue forever" state in a real deployment.
Quoting the ratio backwards. $\frac{R _{I}}{R _{P}} = 30$ means 30 arrivals per 1 processed. Flipping it to $\frac{R _{P}}{R _{I}}$ would look like a healthy margin — the exam-tested habit is to always check which rate is on top and what the number means before reasoning about it.
Mixing units. $R_{I}$ in messages per second and $R_{P}$ in messages per minute give a nonsense ratio. Both rates must use the same time unit before dividing.
Treating a backlog as temporary. A sustained high ratio is not a burst; waiting for the queue to "catch up" never happens while $R_{I} > R_{P}$ holds.

6.1.3 Loss-Sensitive versus Loss-Tolerant Use Cases

Once you accept that back pressure loses messages, the honest question is whether the loss matters. Some use cases can lose messages; others cannot, and the difference is what the rest of the session builds on.

Real-world: credit card transaction messages. A customer performs a transaction; if the pipeline fails to track it, the customer finds out later — "I did not make this transaction" — and you cannot retract what already happened. Every single message matters here.

Real-world: a boiler in a manufacturing unit. A boiler sends messages every minute about pressure, temperature, and density inside the chamber. But you do not need per-minute tracking: any temperature or pressure variance is still visible if you check every five or ten minutes. Monitoring every second adds nothing.

Real-world: weather data. Weather stations stream pressure, temperature, and sky-condition messages continuously. The external ecosystem of the weather does not change drastically in one or two minutes; it changes over fifteen minutes to half an hour. Again, per-minute tracking of every message is wasted effort.

Pitfall — conflating "messages are generated every minute" with "every message must be processed." The boiler and weather cases both generate messages at a fixed periodic interval, yet neither needs all of them. The rate at which a sensor emits data is not the rate at which the business needs data. Asking "how often is data produced?" instead of "how often do I actually need to look?" is the classic route to over-engineering a monitoring pipeline.

So even though messages are generated periodically, the processing engine does not need them at that timescale. Two levers fix an ingestion rate that is far too fast: scale up the processing engine through horizontal scaling (add more nodes), or pick a delivery semantics that says "I do not need every message, I can ignore some." The boiler and weather cases are exactly the situations where the second lever works.

This is where message delivery semantics enters the architecture. The credit-card case cannot afford to lose anything; the boiler and weather cases can afford to lose most of the stream. The rest of the session turns that difference into a formal choice: at most once, at least once, or exactly once.

Recap + bridge. A pipeline whose ingestion rate far exceeds its processing rate ( $R_{I} / R_{P} \approx 30$ ) will flood its queue and lose messages — the only design questions are whether the loss hurts (credit cards: yes; boiler and weather monitoring: no) and what the pipeline may do about it. Handoff: section 6.2 names the three guarantees that answer that "what may we do" question.

6.2 The Three Delivery Semantics

The hook. Suppose a message is lost on its way through the pipeline. Should the system try to send it again? And if it does, what happens when the same message then shows up twice at the other end? The three delivery semantics are the three consistent answers to those two questions — and every streaming system you will ever see implements one of them.

6.2.1 The Three Options at a Glance

There are exactly three message delivery semantics used in streaming systems: at most once, at least once, and exactly once. The name of each one describes the guarantee it makes about how many times a message reaches the consumer.

At most once — a message is sent at most one time; it may never arrive, but it will never arrive twice.
At least once — a message is guaranteed to arrive at least one time; it may arrive more than once.
Exactly once — a message arrives exactly one time: never lost, never duplicated.

6.2.2 At Most Once

With at most once, a message is sent at most one time. If it is lost, it is never resent — the producer simply does not retry. That makes it the cheapest guarantee, and the natural fit for operational monitoring: the chemical factory example and the weather monitoring example both belong here. You send each report once, at its periodic interval; if one report vanishes, the next interval's report is right behind it.

The price you pay is explicit: if too few messages are generated — few transactions, sparse events — losing one is losing a real share of the data, and at most once does nothing about it.

Formalize — what each guarantee actually promises. Let $n$ be the number of copies of a given message that the consumer ultimately receives.

At most once: $n \leq 1$ . The message may be delivered zero times (lost) or one time, but never twice. Loss is possible; duplication is impossible.
At least once: $n \geq 1$ . The message is never lost, but $n$ can be 2, 3, or more if retries deliver copies. Duplication is possible; loss is impossible.
Exactly once: $n = 1$ . The message is delivered and processed one time, no more and no less. Neither loss nor duplication is possible.

The names are the formulas: "at most" caps the count from above, "at least" floors it from below, "exactly" pins it to a single value.

6.2.3 At Least Once

At least once guarantees that every message reaches the processing layer at least one time. If the processing layer does not acknowledge a message, the producer sends it again. The guarantee covers loss, but not duplication: the same message can be delivered more than once, so the analysis layer may receive duplicates and must apply a filter to remove them. It is slightly heavier than at most once, yet far cheaper than exactly once.

The duplicate arises from a timing problem, not carelessness: the producer sends the message, the processing layer receives it and does the work, but the acknowledgement travelling back is lost. The producer cannot tell "your work is done" from "you never received it," so it sends the message again — and the consumer has now seen it twice. The price of never losing anything is occasionally processing the same thing twice; the filter that removes duplicates is the buyer's insurance.

6.2.4 Exactly Once

Exactly once means the message is delivered and processed exactly one time: you lose nothing in delivery and nothing in processing, and you never process the same message twice. It is the strongest of the three guarantees — and the most expensive and hardest to implement, because you must check both ends of the connection to make sure each message was delivered, acknowledged, and processed just once.

Where at least once is a one-sided guarantee (the producer keeps trying until it hears back), exactly once is two-sided: the producer must confirm delivery, and the processing layer must confirm that the work happened exactly once — which in practice means tracking which messages have already been processed so that a retried delivery cannot double-count.

Now that all three are defined, the side-by-side view:

	At most once	At least once	Exactly once
Can a message be lost?	Yes	No	No
Can a message arrive more than once?	No	Yes	No
What happens after a failed send?	Nothing — the message is dropped	The producer sends again	The producer sends again, and the processing layer tracks what was already done
Acknowledgement machinery needed	None	Basic (resend until acknowledged)	Full (acknowledgement plus exactly-once processing checks)
Cost	Cheapest	Middle	Most expensive
Typical fit	Operational monitoring, logging	Analytics, recommendations	Banking, financial transactions

When to pick which: if you can tolerate loss, choose at most once and save the money. If you cannot tolerate loss but can tolerate duplicates, choose at least once and clean up at the consumer. If neither loss nor duplicates are acceptable, you have no choice but exactly once — and you pay for it.

Worked example — three messages, one failure.

Three messages $m_{1}, m_{2}, m_{3}$ are sent in order. During the first send, $m_{2}$ is lost on the network; the producer detects this because it never receives an acknowledgement. What does the consumer see under each semantic?

At most once. The producer does not retry. The consumer receives:

$m_{1}, m_{3}$

Two of three messages arrive. No duplication, but $m_{2}$ is lost forever.

At least once. The producer retries $m_{2}$ . This time it arrives — but the first copy had already been delivered, so the consumer sees:

$m_{1}, m_{2}, m_{2}, m_{3}$

Four deliveries for three messages. Nothing is lost, but $m_{2}$ is delivered twice.

Exactly once. The producer retries $m_{2}$ , and the processing layer recognizes that $m_{2}$ was already processed, so the duplicate is not processed again. The consumer sees:

$m_{1}, m_{2}, m_{3}$

Three deliveries, each exactly once. No loss, no duplication — and the most machinery underneath.

Sense-check: the observed behavior matches the name of each guarantee — the count of $m_{2}$ 's arrivals is 0, 2, and 1 respectively, which are exactly the "at most," "at least," and "exactly" profiles.

Pitfalls.

Treating "delivered" and "processed" as the same thing. A message can be delivered twice but processed once (exactly-once processing), or delivered once and never processed (crash before processing). The guarantees name the contract you buy; check whether the system's contract is about delivery or about end-to-end processing.
Assuming duplicates are free under at least once. Every duplicate travels the full pipeline and consumes processing time; deduplication is a real filter with its own cost. That is why at least once is "cheaper than exactly once," not "free."
Believing exactly once is the default. Most systems ship at-most-once or at-least-once behaviour out of the box; exactly once has to be deliberately configured and paid for. If the notes say a system is reliable by default, check which semantic that reliability actually provides.
Confusing the producer's retry with the guarantee. A producer that retries is implementing at-least-once behaviour at best — retries alone never remove duplicates, they create them.

Where do these three show up in the industry? Monitoring frameworks — the kind that watch a system and raise alerts — commonly run at-most-once, and many even down-sample the stream on purpose, because they can estimate the missing values and because speed matters more than completeness. Financial and billing systems, where every lost record is lost revenue and audit requirements forbid guessing, turn to exactly once through queuing systems. At-least-once sits in the middle as the pragmatic default for analytics: the platform guarantees delivery, and each consumer decides how to handle duplicates in its own application logic — one consumer deduplicates, another uses idempotent operations that are unaffected by seeing the same event twice. The three worked scenarios in sections 6.4–6.6 show each fit in action.

Recap + bridge. At most once never resends and may lose; at least once resends and may duplicate; exactly once resends, deduplicates, and does both ends' bookkeeping — with cost rising in the same order. Handoff: section 6.3 gives the three-question method for choosing among them, and sections 6.4–6.6 run that method on real use cases.

6.3 The Three Decision Questions

The hook. Given any use case — a bank's transaction feed, a shop's browsing stream, a factory's sensor reports — the pipeline must pick one of the three delivery semantics. Three questions, asked in a fixed order, always all three, decide it. This section is the method; the next three sections are the method in action.

For any use case, you decide between the three semantics by asking three questions — always in the same order, and always all three of them.

The decision procedure.

Purpose: turn a use case description into a delivery semantics choice plus a justification.
Input: the use case — what kind of data flows, what happens if it stops, what history must be kept.
Output: one of at most once, at least once, or exactly once, with a reason that cites all three answers.
Steps (never skip, never reorder):

Business impact of a broken channel (6.3.1)
Affordable data loss, in time units (6.3.2)
Storage requirement for historical data (6.3.3)

6.3.1 Question 1 — Business Impact

What will be the business impact if the communication channel between the collection tier and the analysis tier is interrupted? If the connection breaks, data stops flowing and messages are lost. Ask how much that hurts the business: does a missed message mean a lost sale, a false alarm, a regulatory violation, or nothing at all?

The question sorts use cases into two camps before any numbers are involved. A missed credit-card approval is a lost transaction that cannot be redone; a missed recommendation event changes almost nothing about the customer's experience. The first camp will fight for the strongest guarantee; the second can shop by price. Answering this question first forces you to name what the pipeline is actually protecting.

6.3.2 Question 2 — How Much Data Loss Is Affordable

How many days (or how much duration) worth of data can your system afford to lose? The answer can be minutes, hours, days, or "not even a single minute." This is the question that separates banking from monitoring: some systems can tolerate hours of loss, others cannot tolerate seconds.

Notice the unit of the answer: time, not messages. "One minute" is a meaningful figure no matter how fast the stream runs, because the same one-minute gap represents a few events for a quiet sensor and thousands for a busy payment gateway. Express the tolerance in duration first; convert it to message counts only when you need to feel the scale of the loss.

6.3.3 Question 3 — Storage Requirement

What is the storage requirement for historical data? How much history must be kept — the recent past, one month, six months, ten years — and how much does storage actually cost the business? Sometimes this question is decided by regulation, sometimes by strategy.

This question rarely limits the choice of semantics by itself — storage is cheap compared with losing data — but it completes the picture in two ways. First, a settled, mandated retention period removes the storage argument from the debate entirely: if the law requires ten years, the ten years are stored, no further discussion. Second, a short strategic horizon can soften the blow of loss: if old behavior stops mattering after six months, then losing a few recent messages is a small fraction of what the pipeline is allowed to forget anyway.

Pitfalls of the method.

Asking one question and guessing the rest. All three answers must be produced, in order, every time. The exam scenario expects all three questions written down, each answered, with the choice justified from those answers.
Reordering the questions. Impact first, loss tolerance second, storage third. The order is deliberate: impact decides whether loss matters, tolerance decides how much loss is acceptable, storage decides whether the past can be sacrificed.
Answering in message counts instead of duration. "We can lose a few hundred messages" is useless without knowing the stream's rate; "we can lose an hour" is self-contained.
Letting the verdict come first. A pipeline that "just needs exactly once" without going through the questions has not justified anything — the questions exist to force the reasoning to survive scrutiny.
Ignoring regulation. If a regulator has set the retention period, storage is not a cost decision you get to make; pretend otherwise and the answer sheet is wrong before it is written.

Scope of the method. The three questions assume the business can actually answer them — that the impact of loss is knowable and the tolerance is honest. When the answers are unknowable (a brand-new service with no failure history), the safe default is the strongest semantic the budget allows. And when regulation or law settles an answer for you, the method still applies; the question is answered by the statute rather than by the business.

Exam note: a scenario will be given, and you must identify the appropriate delivery semantics by writing all three questions (business impact of a broken collection-to-analysis connection, how many days of data loss is affordable, storage requirement), answering each one, and justifying your choice. There are no marks for naming the semantic alone — the questions and the justification are the answer.

Recap + bridge. Impact decides whether loss matters; the affordable-loss duration decides how much loss is acceptable; the storage requirement decides how much history is sacred — and the three answers, taken together, name the semantic. Handoff: section 6.4 runs the method on transaction data monitoring.

6.4 Worked Example: Transaction Data Monitoring

The hook. A customer swipes a card at a terminal. Somewhere in the bank, a pipeline must record that transaction, understand it, and — for a payment — approve it. Now imagine the pipe between collection and analysis breaks. What does the bank lose? This worked example runs the three questions and lands on the strongest of the three semantics.

6.4.1 The Use Case

The use case is Transaction Data Monitoring: tracking credit card transactions as they happen. Before jumping to a verdict, run the three questions one by one.

The analysis tier does three jobs with this stream: it understands customer spending patterns, it examines the nature of each transaction, and it detects whether a transaction is fraudulent. None of those jobs is decorative — the third one decides whether a criminal charge is blocked. And for a transaction that approves a payment, the stakes are absolute: if the message is missed, the approval itself cannot happen.

6.4.2 Question 1 — Business Impact

Suppose the connection between the collection tier and the analysis tier is lost. Transactions get missed. The impact is substantial — quite significant. The analysis tier uses the data for several things: understanding customer spending patterns, analyzing the nature of each transaction, detecting whether a transaction is fraudulent. And for a transaction that approves a payment, missing the message is a showstopper — the approval itself cannot happen. The general principle in bank applications:

Q: In general, in bank applications you cannot lose any transactional message — correct?

A: Correct. It leads to business loss, because customers switch over to another bank; the bank loses credibility, and under RBI guidelines a bank that cannot account for its transactions may be block-listed from offering banking services. All of these are possible outcomes of losing a single message.

So question 1 answers itself: the impact is severe. One lost message can mean an unapproved payment, an undetected fraud, a customer who changes banks, or a regulator's black mark — the entire range of consequences flows from a single missed event.

6.4.3 Question 2 — How Much Data Loss Is Affordable

Not even a single minute. Hundreds of transactions happen every minute in a bank; if you lose one minute of data, you have lost hundreds of transactions. One minute, one second — none of it is affordable.

Worked example — what one minute of loss costs.

A busy gateway clears 300 transactions per minute. Convert the loss window into lost messages:

$lost messages = rate \times window = 300 \times 1 minute = 300$

Lose one minute and 300 transactions vanish from the record — 300 charges that cannot be reconciled, 300 fraud checks that never ran, and potentially 300 customers whose next statement says something they did not do. Even a ten-second gap at the same rate costs:

$300 \times \frac{10}{60} = 50 transactions$

The affordable loss is zero: no window size above zero is acceptable.

Sense-check: the arithmetic scales linearly — halve the window, halve the losses — so no matter how the window is shaved, any positive window still destroys real transactions. That is the exact meaning of "not even a single minute."

6.4.4 Question 3 — Storage Requirement

Storage is not a big problem for the bank: historical data is kept for analysis and for product promotions — understanding six months of purchase history is usually enough for that. But financial sectors are highly regulated, and one student contributed the key detail:

Q: Financial sectors are highly regulated, and regulations actually guide a default retention of 10 years — it is called a litigation hold. Even employee emails, even a test email, is stored and maintained for 10 years.

A: You can say 10 years by default. Storage, then, is not a barrier to the strongest guarantee — it is a settled, regulated requirement.

So the answers line up: impact is severe, even one minute of loss is unacceptable, and storage is affordable. Every message is critical.

6.4.5 The Verdict: Exactly Once

Every message must be delivered exactly once. At most once is ruled out because it tolerates loss — exactly what banking cannot accept. At least once is ruled out because it can deliver the same message repeatedly — banking does not need duplicates, and it does not want them. Exactly once is the only fit: no loss in delivery, no loss in processing, no duplication.

The trade-off to remember: exactly once is the most expensive and the most difficult of the three semantics to implement. Because you must guarantee that a message is delivered and acknowledged exactly one time, you need multiple checks at both endpoints — the collection side and the processing side. It is expensive, but banking applications require it. The three answers made the decision, and the price tag is the consequence, not a reason to shop around.

Notice how the verdict followed the method. Question 1 said the impact is severe; question 2 said even a minute of loss is unaffordable; question 3 said storage costs nothing that could argue back. Two of the three answers (impact, loss tolerance) point straight at the strongest guarantee, and the third (storage) raises no objection — so exactly once wins before the implementation cost is even considered.

6.4.6 Student Questions and Answers

Q: No system can guarantee that it will never fail; some possibility of failure and connection loss is bound to happen — we cannot predict it or promise a hundred percent that failure will not occur. Assuming we go with high availability and system redundancy, how will exactly once actually work?

A: Exactly once works in two halves. On the delivery side, the collection layer ensures the message is sent — it looks for an acknowledgement from the processing layer, and until the acknowledgement comes, the message is not considered delivered. On the processing side, fault tolerance is taken care of by the processing layer itself, because it is a multi-node cluster, not a single machine: data is replicated — the same data is copied and preserved on different nodes — using partitions and a replication factor. If one node fails, the other nodes still hold the data. So a failure inside the processing layer never loses a message that was already received; the message lives on in the replicas.

The exchange is worth unpacking, because it answers "how can a guarantee exist in a world where everything can fail?" The guarantee is not magic — it is the sum of two engineered halves. The collection half makes delivery verifiable: a message is not "sent" until the processing layer acknowledges it, so an unacknowledged message is retried rather than assumed done. The processing half makes storage survivable: a cluster with, say, a replication factor of 3 keeps three copies of the data on different nodes, so a single node's death costs nothing. The guarantee holds because every individual link in the chain is checked, and every piece of state has a backup. That is the price that the "most expensive" verdict names: each check and each replica is part of the bill.

Pitfalls.

Choosing at most once for a bank "because it is cheap." At most once tolerates loss, and banking tolerates none — the two positions contradict each other, and the contradiction shows up the first time a transaction goes missing.
Choosing at least once for a bank "because it prevents loss." At least once does prevent loss, but it delivers duplicates, and duplicates in transaction records mean double-counted charges, reconciliation headaches, and regulators who ask questions. Banking wants neither loss nor duplicates.
Confusing "the system is redundant" with "the guarantee is free." Redundancy is part of how exactly once survives failures — it does not remove the need for acknowledgements, checks, and deduplication bookkeeping.
Forgetting which endpoint must do which job. Delivery verification lives on the collection side; replica-based fault tolerance lives on the processing side. Swapping the two — or assuming one covers the other — leaves a hole in the guarantee.

Recap + bridge. Transaction data monitoring answers the three questions as severe impact, zero loss tolerance, and regulated ten-year storage — and the only fitting semantic is exactly once, expensive because it checks both endpoints and replicates data across the cluster. Handoff: section 6.5 runs the same three questions on a use case where the answers come out very different.

6.5 Worked Example: Recommendation System

The hook. A visitor browses a store's website: looks at a phone, adds a case to the cart, returns the next evening. A recommendation engine wants to learn from every one of those moves. Now imagine the pipe to the analysis tier dropping a few events — does the business even feel it? This worked example shows how the three questions can land on a middle ground.

6.5.1 The Use Case

Someone visits a website, and you want to build a recommendation system based on user purchase behavior — what the user browsed, what they bought, what they are likely to buy next. Think about the three questions before reading the verdict; the students were asked to reason it out, and their answers are the lesson.

6.5.2 Applying the Three Questions

Question 1 — business impact. If the connection breaks and messages are lost, what happens? One student's read: loss of data is still okay, because this is just a recommendation system — there is no financial or other impact. That is correct: a missed browsing event does not produce a wrong charge or a missed approval; at worst, a recommendation is slightly off.

Q: I think loss of data would still be okay, I guess, because you are not going to lose anything real — it is just a recommendation system, so there is no financial or any other impact.

A: Correct. The loss might not impact the business.

Question 2 — how much loss is affordable. Quite a lot. Sampling is enough: you do not need every event, you need a representative picture — different types of customers, volumes of purchase, patterns of behavior. A sample of the stream supports that.

The reasoning behind "a sample is enough" is worth making explicit. The goal is not to replay every click — it is to learn the shape of demand: which kinds of customers browse, how much they buy, and in what patterns. Those shapes survive sampling. One shopper's missing page-view changes almost nothing about the aggregate picture, so the system can trade completeness for cost.

Question 3 — storage requirement. Storage is not a problem, but you only keep about the past six months, because the platform's business strategy keeps changing. You do not store indefinitely — old behavior is not the strategy anymore.

6.5.3 The Verdict: At Least Once

The right choice is at least once delivery semantics. At most once has a real downside here: if too few transactions happen in a period, you lose data that you did want. At least once costs little — you do not even need the message-received acknowledgement machinery of exactly once — and its only side effect is duplicate data arriving at the analysis step, which you handle with a filter that removes duplicates. Duplicates are cheap to clean; lost data is not. So: at least once, with deduplication at the analysis layer, and six months of storage.

Worked example — duplicates are cheap, loss is not.

A shopper's evening generates three events: $e_{1}$ (viewed a phone), $e_{2}$ (viewed a case), $e_{3}$ (added the case to cart). With at-least-once delivery, a retry delivers $e_{2}$ twice, so the analysis layer receives:

$e_{1}, e_{2}, e_{2}, e_{3}$

The deduplication filter keeps the first copy of each event and drops the rest:

$e_{1}, e_{2}, e_{3}$

The recommendation that follows — "shoppers who viewed a phone also view cases" — is identical to the one the system would have made from the clean stream. The duplicate cost about one line of filter logic; losing $e_{2}$ instead would have silently weakened every pattern built from this session.

Sense-check: with a duplicate, the set of distinct events is unchanged — the filter restores exactly the original three. With a loss, one event is missing from the set and no filter can bring it back. That asymmetry — duplicates are restorable, losses are not — is the whole reason the verdict prefers at least once over at most once.

Pitfalls.

Choosing at most once to save money, then losing sparse data. At most once is tempting for a recommendation stream, but it throws away the same fraction of a quiet Tuesday night as of a sale evening — and on a quiet night, every event is a large share of the data. Sparse periods are exactly where at most once hurts most.
Forgetting that the deduplication filter must exist. At-least-once semantics deliberately allow duplicates; the analysis layer that fails to filter them is not broken, it is unconfigured. The filter is part of the design, not an afterthought.
Reading "at least once" as "deliveries are exact." The name promises every message arrives — nothing about how many times. A system that re-sends on missing acknowledgements is doing its job when duplicates appear.
Sampling and deduplication are different operations. Sampling chooses which events to drop; deduplication restores events that were sent twice. Blurring them leads to a pipeline that either throws away useful data or keeps useless copies.

Recap + bridge. The recommendation system answers the three questions as low impact, generous loss tolerance (sampling suffices), and a six-month strategic horizon — so at least once fits: no loss, duplicates removed by a filter at the analysis layer, and the cheapest machinery that still delivers everything. Handoff: section 6.6 starts a third scenario, operational monitoring, where the same questions pull in two directions at once.

6.6 Worked Example: Operational Monitoring

The hook. A gas chamber in a chemical facility sends a report every minute: pressure, temperature, composition. Is losing one of those reports a small thing or a catastrophe? The honest answer is "it depends" — and this worked example shows why that single word makes operational monitoring the trickiest of the three scenarios.

6.6.1 The Use Case

The third scenario is operational monitoring: a chemical chamber (or weather instrumentation) sends a report message every minute. The discussion starts the three-question analysis, and the class splits into two views before the hour ends.

The tension sits in the phrase "or weather instrumentation." A weather station and a chemical chamber both emit one report per minute, and both are "monitoring." But the weather station's reports describe a system that tolerates delay, while the chamber's reports describe a system where delay can mean danger. The same delivery semantics applied to both would be wrong for one of them — which is why the use case must be examined on its own terms.

6.6.2 The Discussion So Far

Question 1 — business impact. The impact of losing some gas chamber messages depends on the application. If the monitoring is watching for a dangerous situation, losing messages becomes critical — a missed spike could mean a missed alarm. If it is just reporting — routine logging of conditions — a few minutes of loss is acceptable.

Question 2 — how much loss is affordable. The students weighed in:

Q: If it is just some kind of reporting, or not something very critical, I think loss should be okay — a few minutes of loss should be okay.

Q: For normal temperature monitoring it will be fine. But for a chemical plant it will be difficult to lose the data.

A: It depends on the application to some extent, right? A dangerous situation makes the monitoring critical; routine reporting can afford a few lost minutes.

The professor's answer accepts both student views and gives them their boundary: the application decides. Routine reporting can afford a few lost minutes; a dangerous situation cannot afford a missed spike. The difference is not in the data — the same pressure reading is involved — but in what the reading is used for. A value that only feeds a log is expendable; the same value feeding an alarm is load-bearing.

Worked example — the method in progress.

Run the three questions on the chemical chamber and record what has been settled so far.

Question 1 — business impact. Conditional: a missed message is critical when the monitoring guards a dangerous situation (a missed spike could mean a missed alarm); a few lost minutes are acceptable for routine condition logging.

Question 2 — affordable loss. Conditional in the same way: "a few minutes" for reporting; "difficult to lose" for a plant where missing data could hide a hazard.

Question 3 — storage requirement. Not yet discussed in the session.

Status: the two questions answered so far both split the use case into two regimes — routine reporting versus safety-critical monitoring — and the two regimes will pull toward different semantics. The verdict that resolves the split is deliberately left open for a later class, together with one more example.

Sense-check: the partial trace is consistent with the method from section 6.3 — impact first, then loss tolerance — and it already shows the key lesson of this scenario: the same use case can produce two different answers, depending on what the monitoring protects.

Pitfalls.

Answering "monitoring" instead of the scenario. "Operational monitoring is loss-tolerant" is a slogan, not an analysis. The same label covers a weather logger and a safety alarm; the questions must be answered for the specific application.
Treating every report as equally valuable. In safety-critical monitoring the recent reports are worth more than the old ones — a spike two minutes ago is the alarm signal, while last week's readings are history. The value profile of the data is part of the impact.
Assuming "reporting" and "alarming" use the same pipe. If the routine reports and the danger alarms ride the same channel, the channel must satisfy the stricter of the two — losing a routine report is tolerable only while the alarm path is intact.
Letting the word "minutes" answer the question. "A few minutes of loss is okay" is only a complete answer when the stakes are named: minutes of reporting loss versus minutes of alarm loss are different quantities.

6.6.3 Where the Discussion Stopped

The discussion stops mid-analysis and picks up next session with one more example — deliberately, because the instructor announces the exam plan upfront. The two-regime split was the intended lesson, and the completed verdict would follow from resolving it. The class ends with the method running, not finished.

Exam note: a question on this pattern is coming. For a given scenario you will be asked which delivery semantics fits, and you need to write all three questions, answer each, and give the justification. The operational monitoring example is exactly the kind of case that question will use, so expect the completion of this example next class.

Exam note: a question on this pattern is coming — you will be given a scenario, must write all three questions, answer each, and justify the semantics. Operational monitoring is exactly the kind of case the question will use; the completion of this example is expected next session.

Recap + bridge. Operational monitoring splits into two regimes: dangerous situations make every message critical, while routine reporting can afford a few lost minutes — and the method leaves the verdict for the next class on purpose. Handoff: section 6.7 turns from theory to course mechanics — quiz schedule and marks.

6.7 Evaluation Components and Quiz Plans

The hook. Ten questions, fifteen minutes, five marks, one attempt. The quiz sounds small — until you notice that it opens tomorrow morning. This section pins down the quiz mechanics, the full marks structure, and the schedule the class negotiated on the spot.

6.7.1 Quiz 1 Mechanics and Schedule

Quiz 1 was negotiated with the students and finalized at the end of the discussion. Mechanics: 10 questions, 15 minutes, single access with a time bound (you cannot retake it), objective type, worth 5 marks — each question half a mark.

Schedule: enabled the next morning (Monday, 26 February) and available through the following week — students asked for the weekend (the first weekend of March was proposed, but the final agreement was to publish immediately and keep it open from Monday until the following Tuesday). The instructor wanted student consensus before publishing, which is why the announcement went up only after this discussion.

Q: How many days do we get for the quiz? Will it be multiple access or single access with a time bound?

A: Single access with a time bound: 10 questions in 15 minutes. It opens tomorrow — Monday — and stays available until next week Tuesday, so you have the whole coming weekend. Not two weeks, but definitely the weekend.

The mechanics are worth translating carefully, because each one is a rule you must plan around. "Single access" means the quiz can be opened exactly once: no second attempt, no resume after a crash mid-question — the attempt is consumed the moment it starts. "With a time bound" means the clock runs from the start: 10 questions in 15 minutes is 90 seconds per question, which sets the exam strategy — the questions are objective type (no long answers), so the speed problem is reading fast and moving on, not writing. And "half a mark each" is the arithmetic that makes 10 questions worth exactly 5 marks.

6.7.2 The Marks Breakdown

The arithmetic was written out on screen to settle a question about the total:

Worked example — where the 100 marks live.

The five components sum to 60, written out in the session's order:

$5 (Quiz 1) + 10 (Assignment 1) + 30 (Mid-sem) + 10 (Assignment 2) + 5 (Quiz 2) = 60$

and the remaining 40 belongs to the comprehensive exam:

$60 + 40 = 100$

Each quiz is 10 objective-type questions with half a mark per question:

$10 \times \frac{1}{2} = 5$

The complete map: Quiz 1 = 5, Assignment 1 = 10, Mid-sem = 30, Assignment 2 = 10, Quiz 2 = 5, Comprehensive exam = 40, total = 100.

Sense-check: the quiz arithmetic closes the loop — two quizzes at 5 marks each reproduce the two 5s in the component sum, and the component sum plus the 40 reproduces the 100, so no mark is double-counted or missing.

So the two quizzes are 5 marks each, the two assignments 10 marks each, mid-sem 30, and the comprehensive exam 40. One open item: the instructor said they will check with the team whether the comprehensive should be 50 — if so, the other components get scaled accordingly. Mid-sem stays at a minimum of 30 regardless.

The open item is worth understanding as a scaling rule: if the comprehensive becomes 50, the current components (60 in total) cannot simply stay — they would sum to 110. The stated plan is that "the other components get scaled accordingly," with mid-sem protected at its minimum of 30. So the concrete numbers above are the current contract, and the only certain floor is mid-sem ≥ 30.

The session also settled the relationship between these marks and preparation: exams ask only topics covered in class — nothing outside the syllabus — so attendance is the best preparation, and the time set aside for recordings never actually happens. The quiz syllabus, in other words, is the covered-topic list from section 6.9.

Exam note: Quiz 1 runs 10 objective questions in 15 minutes for 5 marks (half a mark each), single access with a time bound, no retakes; it opened Monday 26 February and closed the following Tuesday. Marks structure: 5 + 10 + 30 + 10 + 5 = 60 with compre 40, total 100; a move of compre to 50 would scale the other components, with mid-sem held at a minimum of 30. Exams cover only what was covered in class — attendance beats recordings as preparation.

6.8 Assignments 1 and 2

The hook. Two assignments, one submission, one deadline: one part is a written exploration of a streaming architecture, the other is a working Spark implementation. This section sets out what each part needs, how the groups work, and the one warning the instructor repeats: start now.

6.8.1 Assignment 1 (Part A): Architecture Exploration

Assignment 1, Part A, is worth 10 marks and is an exploration of a streaming architecture. Two architectures were handed out — one architecture per group. You visit the website provided in the announcement, explore the architecture on your own, and produce an article or thesis-style write-up of the concept — the concept, and around the concept, in and out. No more resources are needed: the provided site is the authentic resource, each platform's own web page, where the user documentation lives. Parallel sites are allowed once you start, but the given documentation site is the starting point. The task can be summarized as: go through the website, explore yourself, and come out with a document.

The submission format is flexible: a PPT is fine, a PDF is fine, a Word document is also accepted. Part A requires no new learning — everything in it is architecture and component content already covered in class, and the foundation is enough to finish it within the month.

6.8.2 Assignment 2 (Part B): Spark DataFrame API

Assignment 2, Part B, is worth 10 marks and is the implementation piece: you use the Spark DataFrame API, create an account on the platform, and build the practical solution. The necessary inputs are given after the mid-sem exam, and working on it before that is neither required nor expected.

Part B is deliberately sequenced after mid-sem: the inputs arrive only once the mid-sem window (7–15 March) has passed. That makes Part B a fixed-duration sprint — its window starts when the inputs appear, so its end is dictated by the calendar. This is precisely why Part A exists as an earlier, independent workstream: the two parts are sequential by design, and the month before mid-sem belongs to Part A.

6.8.3 Groups, Submission Format, and Deadlines

The class was split into two groups (six students each, sized so that dropouts do not sink a group). The deadline for both parts is 7th April 2024 — a Sunday, one week after the mid-sem makeup exam — with a common submission: the two parts are submitted together, as a single file, in one submission.

Q: Do all group members have to submit the same document?

A: No — there is one single submission, a presentation containing all the ID numbers and names of the group. If an ID number or name is missing, that student does not get marks: a missing ID is understood to mean that person contributed nothing. That is why you must discuss and make sure every member's ID and name is on the submission.

Q: What exactly do we need to do in this assignment?

A: Two architectures have been given — one architecture for one group. Go to the website and explore your architecture, keeping whatever points we discussed in class. You can think of it as writing an article or a thesis on the concept.

Q: Should the format be a PPT or a PDF?

A: You can make a PPT. A Word document is also accepted — the point is the discussion and the content, not the format.

Q: About the deadline — submitting both parts by 7 April is tight, because we might have other things to do, like a makeup for another subject.

A: Start Part A now and finish it early. You have the foundation for it, so it can be done within this month — an internal target of 3 or 9 March works. Part A should be finished in the first week of March, before the mid-sem (7–15 March). Part B is started after mid-sem, when the inputs arrive. If 7 April turns out to be genuinely hard, an extension of one week is possible — but do not bank on the deadline: start now.

The warning behind all of this is a real one: in another subject last semester, the assignment window was similar — 40 to 45 days — and there was plenty of time, so work was put off until after makeup, after the comprehensive, until one week before the deadline. It turned out to be an ocean — not manageable in one week. Night outs, and still not 100% complete. The lesson: start Part A this month, not in April.

Pitfalls of assignment planning.

Missing one name on the submission. The submission is the only evidence of contribution: a missing ID or name means no marks for that person. The final check before submitting is a read of the name-and-ID list, not a re-read of the content.
Leaving Part A for after mid-sem. The calendar says Part B starts only after the inputs arrive — so Part A cannot be "done later" without compressing both. The internal target is the first week of March; the deadline itself is the last resort, not the plan.
Treating the extension as a plan. A one-week extension is possible on request, but relying on it is gambling on a lenient outcome. The instructor's own story — a 40- to 45-day window that still ended in night outs and incompleteness — is the evidence that long windows do not survive procrastination.
Splitting the write-up into last-week essay writing. "Article or thesis style" means the write-up needs to be shaped continuously, not poured out in one sitting; the exploration and the writing are one activity.

Exam note: both parts are due 7th April 2024 as one submission per group; internal target is Part A finished in the first week of March (3 or 9 March), before mid-sem (7–15 March); Part B starts after mid-sem when inputs arrive; makeup exams give no extra time, so the regular exam is the schedule-compatible path.

Recap + bridge. Assignment 1 explores one of two architectures from its official documentation; Assignment 2 builds the Spark DataFrame solution after mid-sem; the two parts submit as one file on 7th April, with every group member's ID and name listed. Handoff: section 6.9 steps back and maps what the course has covered so far — which is also the quiz territory.

6.9 Course Status and Syllabus Recap

6.9.1 Topics Covered So Far

The session paused to enumerate what the course has covered, which also sets the quiz syllabus:

Big data systems — the opening module.
Streaming architectures — batch processing versus stream processing, and the two umbrella designs: Lambda architecture and Kappa architecture. The reminder worth keeping: Lambda is used for both batch and stream paths in the same system.
Apache ZooKeeper — ZooKeeper's Z-nodes and the node types.
The collection layer and the data flow layer — including the two interaction patterns of the data flow layer: client-server and publisher-subscriber (PubSub).
Rate problems in the data flow layer — problems on the rate of ingestion and the rate of processing, including earlier calculations using the same ratio used in this session, the ingestion-to-processing ratio

$\frac{R _{I}}{R _{P}}$

where $R_{I}$ is the rate of ingestion and $R_{P}$ is the rate of processing. The earlier calculation mentioned in class fits this same formula: it compared how fast data arrives with how fast the pipeline works through it, exactly as the back-pressure discussion did. The garbled "RPE by RI" phrase from the session is this same ratio — the ingestion rate over the processing rate — and nothing else.

The list is worth reading as a path, not a pile. The course started with big data systems as the stage, then built the streaming architectures that live on it, then descended into the coordination machinery (ZooKeeper), then the collection and data flow layers — and now stands at the rate problems of the data flow layer, which is where this session's delivery semantics attach. Every entry except the current one is settled material; the current one is the topic under construction.

6.9.2 What the Recap Means for the Remaining Sessions

Message delivery semantics — the substance of this session — is the new topic that follows directly from the data flow layer and its rate problems. The syllabus review also serves the quiz: the covered list is exactly the quiz territory, and planning the remaining sessions depends on knowing where the class stands. Students were advised to take notes and track this list themselves, because it is the map for everything left to cover.

Recap + bridge. The course has covered big data systems, streaming architectures (Lambda for both batch and stream paths, Kappa alongside), ZooKeeper's Z-nodes and node types, the collection and data flow layers with their client-server and PubSub patterns, and the rate problems of the data flow layer — and the session's own message delivery semantics is the topic that grows directly out of that last entry. That covered list is the quiz territory, so it doubles as the study map.

Exam Guidance Summary

Delivery semantics is a marked question. A scenario will be given and you must identify the appropriate message delivery semantics: write all three questions (business impact of a broken collection-to-analysis connection, how many days of data loss is affordable, storage requirement), answer each, and justify your choice. The instructor announced this pattern upfront: "I am telling you upfront which delivery semantic… you need to put all the three questions, you need to put the justification."
Three worked scenarios to master: Transaction Data Monitoring → exactly once (no loss of even a minute; regulated 10-year retention; most expensive semantic). Recommendation system → at least once (loss tolerable, sampling enough, six months of storage, duplicates filtered at analysis). Operational monitoring → verdict completed next class, because the pattern will be examined.
Quiz format: 10 objective-type questions, 15 minutes, 5 marks, half a mark each, single access with time, no retakes. Quiz 1 opened Monday 26 February and closed the following Tuesday (4–5 March).
Marks structure: Quiz 1 (5) + Assignment 1 (10) + Mid-sem (30) + Assignment 2 (10) + Quiz 2 (5) = 60; comprehensive exam = 40; total 100. The team is checking whether the comprehensive should be 50, with the rest scaled; mid-sem stays at least 30.
Assignment deadlines: both parts due 7 April 2024, submitted as one file per group. Internal target: Part A done by 3–9 March, before the mid-sem (7–15 March). One-week extension possible on request.
Makeup exams are discouraged: they give you no extra time. Taking the regular exam keeps enough time to finish the assignments.
Exams cover only what was covered in class — nothing crazy. Attendance is the best preparation; recordings are not a substitute, because time allocated to them never actually happens.

Exam note: the delivery-semantics question is the centrepiece — a scenario, all three questions answered in order, and a justified semantic. Master the three worked scenarios: banking lands on exactly once (expensive but mandatory), recommendations land on at least once (duplicates filtered at the analysis layer), and the operational monitoring verdict — split between routine reporting and safety-critical monitoring — is completed next class.

Key Industry Applications

Real-world: banking and credit cards drive the exactly-once case — missed transactions mean business loss, customers switching banks, loss of credibility, and RBI block-listing of the bank itself; payment approval is a showstopper that cannot be missed.
Real-world: regulatory retention — financial sectors default to 10 years of retention (litigation hold), so storage is settled by law, not by cost.
Real-world: manufacturing safety — boiler pressure, temperature, and density messages can be sampled at 5–10 minute granularity; routine monitoring tolerates lost minutes, while dangerous-situation monitoring does not.
Real-world: weather data — pressure, temperature, and sky-condition streams change over 15–30 minute horizons, so per-minute tracking is wasted; at most once fits.
Real-world: recommendation systems — purchase-behavior streams tolerate sampling and duplicated events (deduplicated at analysis), with about six months of stored history before strategy changes make it irrelevant.
Real-world: back pressure engineering — when ingestion is ~30 times processing speed, queue flooding loses messages; the two responses are horizontal scaling of the processing engine or a lighter delivery semantics.

Takeaway: the same architecture decision — which delivery semantics to buy — recurs across industries, and the deciding factor is always the same three answers: how much a missed message hurts, how long a gap the business can absorb, and how much history must survive.

SPA Lecture 06 notes · Message Delivery Semantics

Stream Processing and Analytics· postgraduate· 2026-08-07

Sections Breakdown

16.1 Motivation: When Ingestion Outruns Processing

Why a pipeline whose ingestion rate dwarfs its processing rate floods its queue and loses messages, and which use cases can tolerate that loss.

26.2 The Three Delivery Semantics

At most once, at least once, and exactly once - what each guarantee promises about loss and duplication, and the cost that rises with it.

36.3 The Three Decision Questions

The fixed-order method for choosing a semantic: business impact of a broken channel, affordable data loss, and the storage requirement.

46.4 Worked Example: Transaction Data Monitoring

The three questions run on bank transactions: severe impact, zero loss tolerance, regulated ten-year storage - exactly once.

56.5 Worked Example: Recommendation System

The three questions run on purchase-behavior streams: low impact, sampling-tolerant loss, six-month history - at least once with deduplication.

66.6 Worked Example: Operational Monitoring

The scenario that splits in two: safety-critical monitoring versus routine reporting, with the verdict deliberately deferred.

76.7 Evaluation Components and Quiz Plans

Quiz 1 mechanics and the full marks structure: 10 questions in 15 minutes for 5 marks, and how the 100 marks are composed.

86.8 Assignments 1 and 2

Part A's architecture exploration and Part B's Spark DataFrame implementation - the 7 April deadline and the single group submission.

96.9 Course Status and Syllabus Recap

What the course has covered so far - the list that doubles as the quiz territory.

10Exam Guidance Summary

The professor's exam strategy: the delivery-semantics question pattern, the three scenarios to master, and the quiz and assignment calendar.

11Key Industry Applications

The delivery-semantics decision in the wild: banking, regulatory retention, manufacturing safety, weather data, recommendations, and back pressure.

Postgraduate students in Stream Processing and Analytics

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

6.1 Motivation: When Ingestion Outruns Processing

Must-know: Back pressure: when R_I/R_P reaches the order of 30, one processed record leaves ~30 waiting, the queue floods and messages are lost; fix with horizontal scaling or a lighter delivery semantics.

$\frac{R _{I}}{R _{P}} \approx 30$

⚠️ Top pitfall: Assuming the queue is infinite; quoting the ratio backwards; mixing time units between R_I and R_P.

Self-check: If R_I = 300 messages/min and R_P = 10 messages/min, how many messages arrive while one is processed?

Connects to: 6.2

6.2 The Three Delivery Semantics

Must-know: At most once: n<=1 (loss possible, duplication impossible). At least once: n>=1 (loss impossible, duplication possible, filter at analysis). Exactly once: n=1 (delivered and processed exactly one time, most expensive).

$n \leq 1 vs n \geq 1 vs n = 1$

⚠️ Top pitfall: Treating 'delivered' and 'processed' as the same thing; assuming duplicates are free under at least once; believing exactly once is the default.

Self-check: Under at least once, one failed send of m2 produces what sequence at the consumer?

Connects to: 6.1, 6.3

6.3 The Three Decision Questions

Must-know: Always ask all three questions in order: business impact of a broken channel, affordable loss duration (minutes to days to 'not even a single minute'), and storage requirement; the exam scenario requires all three written, answered, and the semantic justified.

⚠️ Top pitfall: Asking one question and guessing the rest; reordering the questions; answering in message counts instead of duration; letting the verdict come first.

Self-check: What are the three questions, and why must the storage question come third?

Connects to: 6.4, 6.5, 6.6

6.4 Worked Example: Transaction Data Monitoring

Must-know: Banking = exactly once: severe impact (payment approval is a showstopper), no minute of loss affordable, 10-year regulated retention; at most once tolerates loss and at least once delivers duplicates, both unacceptable; exactly once works via acknowledgement at collection plus partitions and a replication factor in the processing cluster.

$lost messages = R \times window$

⚠️ Top pitfall: Choosing at most once 'because it is cheap' for a bank, or at least once 'because it prevents loss' - banking needs neither loss nor duplicates.

Self-check: Why is at least once rejected for transaction data monitoring even though it prevents loss?

Connects to: 6.3, 6.5

6.5 Worked Example: Recommendation System

Must-know: Recommendation system = at least once: no financial impact, sampling is enough (representative picture of customers and purchase patterns), six months of storage before strategy changes; at most once would lose sparse-period data, duplicates are removed by a filter at the analysis layer.

⚠️ Top pitfall: Choosing at most once and losing sparse data; forgetting the deduplication filter; confusing sampling (choosing what to drop) with deduplication (restoring what was sent twice).

Self-check: If at-least-once delivery produces e1, e2, e2, e3, what does the dedup filter emit?

Connects to: 6.4, 6.6

6.6 Worked Example: Operational Monitoring

Must-know: Loss tolerance depends on the application: a dangerous situation makes monitoring critical (a missed spike could mean a missed alarm), routine reporting can afford a few lost minutes; the verdict is completed next class because a question on this scenario pattern is planned.

⚠️ Top pitfall: Answering 'monitoring' instead of the scenario; treating every report as equally valuable; assuming reporting and alarming use the same pipe.

Self-check: Why do two student answers - 'a few minutes of loss is okay' and 'a chemical plant makes losing data difficult' - both get accepted?

Connects to: 6.5, 6.3

6.7 Evaluation Components and Quiz Plans

Must-know: Quiz 1: 10 objective questions, 15 minutes, 5 marks (half a mark each), single access with a time bound, no retakes, opened Monday 26 February until the following Tuesday. Marks: 5 + 10 + 30 + 10 + 5 = 60 with compre 40, total 100; compre may become 50 with the rest scaled; mid-sem stays at least 30.

$5 + 10 + 30 + 10 + 5 = 60, 60 + 40 = 100, 10 \times \frac{1}{2} = 5$

⚠️ Top pitfall: Treating the quiz as multiple-access or untimed; forgetting that single access means the attempt is consumed the moment it starts (90 seconds per question).

Self-check: How many marks is each quiz question worth, and how do the components sum to 100?

Connects to: 6.8, 6.9

6.8 Assignments 1 and 2

Must-know: Both parts due 7th April 2024 as one submission per group; Part A (architecture exploration from the provided documentation site) targeted for the first week of March before mid-sem (7-15 March); Part B (Spark DataFrame API) starts after mid-sem when inputs arrive; missing ID or name on the submission means no marks for that student; one-week extension possible but not to be relied on.

⚠️ Top pitfall: Missing a group member's ID/name on the single submission; leaving Part A until after mid-sem; banking on the one-week extension (the 40-45 day assignment that still became an ocean).

Self-check: When do the inputs for Part B arrive, and why must Part A finish before then?

Connects to: 6.7

6.9 Course Status and Syllabus Recap

Must-know: Covered so far: big data systems; streaming architectures (Lambda serves both batch and stream paths in one system; Kappa alongside); ZooKeeper Z-nodes and node types; collection and data flow layers with client-server and PubSub patterns; rate problems in the data flow layer (ingestion-to-processing ratio R_I/R_P) - this list is exactly the quiz territory.

$\frac{R _{I}}{R _{P}}$

⚠️ Top pitfall: Forgetting that Lambda serves both batch and stream paths in the same system - that is what distinguishes it from a pure streaming design.

Self-check: Which architecture serves both the batch and the stream path in one system?

Connects to: 6.1, 6.2

Exam Guidance Summary

Must-know: Delivery semantics is a marked question: a scenario is given, and you must write all three questions (business impact, affordable data loss, storage requirement), answer each, and justify the semantic; master exactly once (banking), at least once (recommendations, duplicates filtered), and the operational monitoring case completed next class.

⚠️ Top pitfall: Naming the semantic without writing the three questions and the justification.

Self-check: Which three items must every delivery-semantics answer include?

Connects to: 6.3, 6.4, 6.5, 6.6

Key Industry Applications

Must-know: Named real-world anchors: exactly once for banking/credit cards (payment approval is a showstopper; RBI block-listing risk), 10-year litigation-hold retention for financials, 5-10 minute sampling for boiler monitoring, 15-30 minute horizons for weather, six months of history for recommendations, and ~30:1 back pressure with the two levers (horizontal scaling or lighter semantics).

⚠️ Top pitfall: Applying one semantic across industries without asking the three questions - the deciding factor is always impact, affordable loss, and storage.

Self-check: Which two responses fix an ingestion rate far faster than processing?

Connects to: 6.1, 6.4, 6.5, 6.6

Previous LectureLecture 05 · ZooKeeper: Configuration and Coordination in the Streaming Pipeline

Next LectureLecture 07 · Kafka Architecture

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

6.1 Motivation: When Ingestion Outruns Processing

6.1.1 The Streaming Pipeline and the Two Rates

6.1.2 Back Pressure

6.1.3 Loss-Sensitive versus Loss-Tolerant Use Cases

6.2 The Three Delivery Semantics

6.2.1 The Three Options at a Glance

6.2.2 At Most Once

6.2.3 At Least Once

6.2.4 Exactly Once

6.3 The Three Decision Questions

6.3.1 Question 1 — Business Impact

6.3.2 Question 2 — How Much Data Loss Is Affordable

6.3.3 Question 3 — Storage Requirement

6.4 Worked Example: Transaction Data Monitoring

6.4.1 The Use Case

6.4.2 Question 1 — Business Impact

6.4.3 Question 2 — How Much Data Loss Is Affordable

6.4.4 Question 3 — Storage Requirement

6.4.5 The Verdict: Exactly Once

6.4.6 Student Questions and Answers

6.5 Worked Example: Recommendation System

6.5.1 The Use Case

6.5.2 Applying the Three Questions

6.5.3 The Verdict: At Least Once

6.6 Worked Example: Operational Monitoring

6.6.1 The Use Case

6.6.2 The Discussion So Far

6.6.3 Where the Discussion Stopped

6.7 Evaluation Components and Quiz Plans

6.7.1 Quiz 1 Mechanics and Schedule

6.7.2 The Marks Breakdown

6.8 Assignments 1 and 2

6.8.1 Assignment 1 (Part A): Architecture Exploration

6.8.2 Assignment 2 (Part B): Spark DataFrame API

6.8.3 Groups, Submission Format, and Deadlines

6.9 Course Status and Syllabus Recap

6.9.1 Topics Covered So Far

6.9.2 What the Recap Means for the Remaining Sessions

Exam Guidance Summary

Key Industry Applications

SPA Lecture 06 notes · Message Delivery Semantics

Sections Breakdown

Exam Revision Notes

6.1 Motivation: When Ingestion Outruns Processing

6.2 The Three Delivery Semantics

6.3 The Three Decision Questions

6.4 Worked Example: Transaction Data Monitoring

6.5 Worked Example: Recommendation System

6.6 Worked Example: Operational Monitoring

6.7 Evaluation Components and Quiz Plans

6.8 Assignments 1 and 2

6.9 Course Status and Syllabus Recap

Exam Guidance Summary

Key Industry Applications

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat