Events, Topics, Non-Functional Requirements, and the Big Data Data Model

2.1 Data Processing Systems and the Two Core Operations

2.1.1 What a Data Processing System Is

This session looks at system design for big data systems: what data processing applications are, and why they behave differently from traditional applications. Two areas get most of the attention: the requirements of data processing systems, and the data model — traditional versus data intensive applications.

When we say a data processing system, we mean orchestration: orchestration of various tools and integration towards the different practices the organization follows. It is not one single tool that caters to all needs — it is an integrated set of tools working together. Think of a kitchen brigade in a restaurant: the chef does not expect one appliance to wash, chop, cook, and plate. A prep station handles chopping, another station handles cooking, a third handles plating — and the chef coordinates them so that everything arrives at the table at the right time. A data processing system is the same idea: one tool ingests data, another stores it, another analyzes it, and integration code coordinates the flow.

Every application starts from what we call programming systems, and there we talk about data types. The three data types to know are:

• Structured data — data with a fixed, predefined shape: columns and types decided in advance, like a well-filled form. Rows in a relational table are the classic example.
• Semi-structured data — data that has some organization but no rigid schema: a JSON message or an XML document. Every message can carry slightly different fields, and that flexibility is the point.
• Unstructured data — free-form content with no predictable structure: a text document, an email body, an image, a video.

On top of that, any application's requirements fall into two categories:

• Functional requirements — what the system must do. "When a customer swipes a card, send an alert" is a functional requirement.
• Non-functional requirements — how well the system must do it. "The alert must arrive within 200 milliseconds, 99.9% of the time, even when the database is slow" is a non-functional requirement.

The non-functional side gets a deep look in section 2.5, where it splits into reliability, scalability, and maintainability.

2.1.2 The Only Two Stream Computations: Filtering and Aggregation

A bank alerting you about a possibly fraudulent card transaction is filtering: the system watches each event and keeps only the ones that look suspicious. A monthly report that says "10,000 transactions, of which 35% converted into EMIs" is aggregation: many events collapsed into one number.

Keep this pair in mind — it comes back at the end of the session as one of the two primary motivations for the entire big data architecture family.

2.1.3 Traditional Applications and CRUD Operations

Traditional applications — think of any website application, for example a Java application or a .NET application — work with a database through what we call CRUD operations: create, retrieve, update, delete. When you work with typical tables in SQL, you create the table, retrieve information from it, update a particular value in a cell, and delete a table or a row. That is the complete toolbox of traditional data work.

Now translate that toolbox to data intensive applications. Of the four operations, only two survive:

No updates. No deletes. In streaming applications, the data work is event processing, and event processing implicitly involves exactly the two operations from 2.1.2 — filtering and aggregation. So the connection is direct: CRUD minus update and delete, reshaped as filtering and aggregation.

2.1.4 What "Data Intensive" Means

When we say data intensive application, we are talking about performing computations on a regular basis while the data is continuously moving. We want to perform certain calculations on a potentially endless, evolving source of data. The key contrast: it is not that data is at rest — not data stored in a table that sits still. The data is continuously moving, and we capture some of that moving stream and do the analysis on it. Data at rest versus data in motion is the divide.

That reading of the term raised questions in class, and the answers sharpened the definition:

Real-world & domain connection: Every alerting, monitoring, and analytics product you use is a data processing system on this definition. Payment gateways (Razorpay, Stripe) ingest card events and filter for risk; ride-hailing apps aggregate GPS pings into surge-pricing summaries; cloud monitoring platforms (Datadog, Prometheus) filter metrics by severity and aggregate them into dashboards. In each one, an architecture of integrated tools processes moving data with nothing but filtering and aggregation — which is exactly why this course can study the whole family of big data architectures with just those two operations in hand.

2.2 Events: The Fundamental Building Block

2.2.1 The Definition of an Event

In real-time data processing, event and message are used synonymously. The definition to fix in your mind: an event is an immutable fact related to a specific context, occurred in the system. Every later concept in this course builds on this sentence, so it is worth memorizing piece by piece. Unpack it:

• Immutable — once it exists, it can never be changed (section 2.2.3 explains why).
• Fact — it records something that actually happened; it is not an opinion or a future plan.
• Related to a specific context — every event belongs to a situation: a card swipe, a page visit, a message sent.
• Occurred in the system — the event is a record of a happening, captured where the system can see it.

2.2.2 What an Event Contains: Who, Where, When

What is the structure of an event? It includes things like the timestamp, the amount of the transaction, and where the transaction was made. In short: who, where, and when. The bank already knows the who — it knows which card you used — so the event must carry the where and the when, and the who becomes a validation step.

Here is the design habit the instructor emphasizes: at requirement gathering, when you are building the system, you ask the bank "what should the structure of that message be? What should it contain?" That question is exactly what defines the event. The message content is a requirement you extract from the business, not something you invent.

2.2.3 Immutable Facts and Business Outcomes

Why do we call it an immutable fact? Once the transaction is done, there is no way to go back and change it. The message that arrives on your phone cannot be changed — not by you, not by the bank. The only thing the bank can do is analyze: how many transactions happened as part of this offer, and what impact they had.

Real-world: a credit card offer of 10% off on Amazon, offered by a bank such as SBI. The events are all the transactions coming from Amazon. The bank looks at those events and checks the impact of the offer it launched — and note the detail that when the transaction information arrives, the 10% discount is already applied to it; the event captures the world as it was at that moment.

The point of all of this is business outcome. Whatever application you build, the goal is to generate insight by analyzing these events. Real-world: after a festival season such as Dasara, the bank analyzes what offers it gave, how much transaction volume it saw, and how many of those transactions converted into EMIs (because these offers are given on EMIs), and then revisits its offers and plans for the next season based on what it has seen. That decision making is the expectation from these systems — there is a business goal attached. We are not doing stream processing as a fancy exercise; everything is connected to the business outcome.

2.2.4 Student Questions and Answers

Real-world & domain connection: The event is the atomic unit of every modern data platform. Payment rails, IoT sensors (each temperature reading is an event), clickstreams, ad impressions, and ride-hailing trip records are all streams of lightweight events carrying who/where/when. Kafka and other message brokers are built entirely around this definition: they store and move immutable event records while analytics systems (Spark, Flink) do the only two operations from 2.1.2 on them.

2.3 Infinite Tables, Batch vs Stream, and Message Formats

2.3.1 The Infinitely Growing Table

When you store events, they become individual rows in a database table — but this table is special. It is an infinite table: an infinitely growing table. Event E1 sits in one row; when the second event comes, E2 sits in the next row; E3 after that. The table is infinitely long, and the only write operation is append — you only append the record. You never update a row and never delete a row.

How do you get value out of an append-only table? After a number of records are added, you collect them as a single batch. Then you look at the whole batch and compute batch-level numbers. Real-world: you do not look at "amount" per transaction; you look at the volume of transactions in the batch, the percentage of transactions converted into EMI, and you segment by what type of product was purchased (electronic gadgets versus other purchases, say a gas stove), by vendor, and by whether EMI conversion happened. That segmentation feeds analytics and dashboards.

So the picture of this first type of application: processing is filtering and aggregation to generate dashboards; data is stored in an infinitely long table where you continuously append at the end; and the other operation is selecting from the table as a micro-batch or mini-batch to generate insights.

2.3.2 Batch Processing vs Stream Processing

That first picture is one type of application: the messages land in the table, accumulate, and later the system analyzes them as one batch — this is batch processing. The second type: the messages are generated on the fly and you perform the analysis as they arrive — that is stream processing. Both are fundamental, and both get examined in detail in later sessions; what matters now is the two shapes: analyze what has accumulated, or analyze what is arriving.

2.3.3 What a Data Stream Is

A data stream is essentially a stream of immutable data. Each event is on the order of 1 KB — these are very lightweight events. Streams also vary in rate: sometimes the stream carries 100 records in a particular interval, sometimes 50 records in a similar interval; there is no fixed cadence.

The "order of a few KB" is the important point, and it is not arbitrary. When you look at the messages you get on your mobile phone during transactions, the size is on the order of KB — it cannot be huge. The size matters for processing: on fast-moving data, when you want a small window — say a window of five events, or a window of ten events — those messages have to fit in memory so they can be processed quickly. If the messages are voluminous, neither small windows nor quick processing is possible.

2.3.4 Message Formats

What does a typical message look like? Usually something like a JSON file: transaction number, amount, and so on. Whenever something happens as an event, these messages are sent to the application, and you take the sequence of events and process them to get more meaningful data. JSON is the common picture, but it is not the only format — there are multiple formats: AVRO, plain TXT file format, XML, and others. You will see this concretely when the course goes deep into PySpark.

{
  "transaction_id": "TX-77821",
  "card_id": "1234-XXXX-5678",
  "amount": 2400,
  "merchant": "Amazon",
  "timestamp": "2026-08-06T14:32:11Z"
}

2.3.5 Student Questions and Answers

Real-world & domain connection: This section is the blueprint of real streaming platforms. Kafka stores exactly this way — an append-only, immutable log that every consumer reads sequentially; Spark's streaming mode reads micro-batches off it. The six-month retention policy is how real banks operate (dispute resolution, regulatory audit), and message formats map directly to real choices: AVRO is Kafka's default serialization in most production pipelines, JSON for quick integrations, and plain text for lightweight log ingestion.

2.4 Topics: The Logical Grouping of Events

2.4.1 The Email Subject Line Analogy

Here is a question worth asking yourself: in an office, do you read all the emails you receive? In good practice you should, but practically — no, you don't, and it is not necessary. On what basis do you decide? You look at the subject line — that is the first thing. The second thing is whether you are in the To list or the CC list.

2.4.2 Topics Around Business Goals

Under the topic "analyzing fraudulent transactions," you have the messages that concern fraud. Under another topic named "customer segmentation," the corresponding messages land. Depending on the logical grouping context you attach to the events when they are generated — that many topics you create. The grouping is driven by the business question you need to answer.

2.4.3 Why Events Stay Lightweight

Within topics, messages stay very lightweight because you ensure they contain only the information that is supposed to be there at the moment the transaction happened. Real-world: the bank does not send you a photo of you making the transaction along with the alert. Why? Because it would needlessly increase the size of the message and it does not add any business value.

2.4.4 Student Questions and Answers

Real-world & domain connection: Topics are the core abstraction of Kafka — a named, partitioned stream of events that producers write to and consumers subscribe to; Kafka topics such as "transactions", "fraud-alerts", and "customer-activity" are exactly the business-goal groupings described here. The same idea appears in Apache Pulsar, AWS Kinesis (shards as logical groupings), and messaging platforms like RabbitMQ exchanges, which is why this conceptual section is the foundation for the tool-heavy sessions that follow.

2.5 Non-Functional Requirements: Reliability, Scalability, Maintainability

Associated with these applications are three non-functional requirements: reliability, scalability, and maintainability.

2.5.1 Reliability: Coming Out of Failure

Reliability is the ability of the system to come out of failure. Failure can happen because of a fault. When it does, you have options. If a link between components breaks — say a network failure — you may eventually lose messages. Then you decide: either it is okay to lose some messages, or you cannot afford to lose them, and you use a backup plan called replication — you create a copy of the messages on some other machine. That backup plan is how reliability is taken care of.

2.5.2 Fault versus Failure

There is an important vocabulary distinction. Failure refers to the system. Fault is with respect to a component — a column, a database table, or any single component. When you build an architecture to process events, that architecture has many touch points: you need a data ingestion mechanism to bring data in; once data is ingested you need a mechanism to send the messages to the processing unit; and in the processing unit you must keep in mind the rate at which messages arrive versus the rate at which the processor processes them. It is those touch points that we refer to when we talk about faults and failures.

2.5.3 Scalability: Horizontal and Vertical

Scalability is the ability of the system to cope with upcoming load. There are two ways to achieve it. Vertical scalability: you have a single machine and you keep increasing its RAM and processing power. Horizontal scalability: you take replicas of the machine and create a cluster — you add more and more similar machines and set up a mechanism for communication among them, so the machines work together in a synchronized manner.

When to pick which: vertical is the simplest start for predictable load; horizontal is the direction for event-driven systems, because message rates grow without warning and adding machines is how the system absorbs that growth.

2.5.4 Maintainability: Operability, Simplicity, Extensibility

Maintainability is adaptability to change: when change requests come, how well the system adapts; how easy it is to extend the existing system to something else. Maintainability is looked at through three aspects:

• Operational aspects: automation and integration with other tools — how smoothly the system operates in operations.
• Simplicity: balancing the features and functionalities you want against the components and interfaces you need. You may want a lot of features, but are you increasing the number of components? If the system becomes complex, it is not maintainable.
• Extensibility: the ease of accommodating new requirements — and requirements keep changing from time to time, so this matters constantly.

2.5.5 Student Questions and Answers

Real-world & domain connection: These three words are the vocabulary of every production data platform. Netflix's Chaos Monkey deliberately kills machines to prove the system comes out of failure — the operational embodiment of reliability testing. Cloud providers sell horizontal scaling directly (auto-scaling groups add machines as load rises). And Kafka's design — an append-only log, replicated across brokers — is a textbook realization of all three requirements at once: replicas give reliability, partitioned clusters give horizontal scalability, and the simple append model keeps the system maintainable.

2.6 The Page-Hits Use Case: From Updates to Shards

2.6.1 The Traditional Design: Server-Side Updates

Here is a worked design scenario that shows why big data systems exist. We are designing an application to monitor page hits — the number of visitors to a portal — and to generate analytics on those counts. Every time a particular user visits the page, we need to increment the count.

How does the count get updated in the traditional design? The server performs server-side updates, and the database gets updated in periodic intervals. On the left you have many clients interacting with your server. The message in this use case is a simple count: every time a visitor comes in, you increment the count. That works fine — until the website becomes popular.

2.6.2 The Queue: Absorbing Rate Mismatch

When the site becomes popular, many users visit concurrently, and every visit increments the count. The number of page hits grows, and the database writes cannot keep up with the rate at which the messages arrive. The first fix: maintain an intermediate queue between the web server and the database, so the rate mismatch — between the database's writing ability and the rate at which messages are coming in — is absorbed. Messages wait in the queue instead of being dumped straight into the database.

2.6.3 Database Shards for Parallel Writes

At some point even the queue is not enough — you are still not able to meet the speed. The design moves from updates to data partitions, which in database terms are called database shards. A shard is a partition of the data, distributed across a database cluster. You create multiple tables to store this information because the site now has millions and millions of visitors, meaning millions and millions of transactions.

The point of sharding: each time a visitor comes in, a message describing that visit is written to the table. With more people, you need more update operations. If you can write to several shards in parallel, you spread those write operations across machines and improve the latency mismatch. It is not that each shard holds a copy — each shard holds different data; the writes go in parallel.

2.6.4 The Fault Tolerance Wall

But even this becomes a bottleneck at some point, because of issues about fault tolerance. Every shard holds different data — if something goes wrong, you lose that data. Why not simply keep multiple copies of each shard? Because you are already struggling to manage the pace at which messages and events are coming in — that is your utmost priority. If you add copying, you need backup mechanisms for everything, and managing all those shards and copies makes life more complex. It is not a straightforward thing to manage, and repartitioning the shards brings a whole bunch of complications.

2.6.5 The Motivation for a New Ecosystem

The question that drives everything: "When I am doing these kinds of frequent updates to the tables, is there any better mechanism than what I am doing here?" That is the main motivation of data processing in big data systems. The evolution of big data systems helps in multiple ways: the data model itself is different, and the setup with respect to fault tolerance is different.

At the end of the session, the instructor distills this into the two primary motivations:

1. The nature of the data and the operations you do on it — the operations are only filtering and aggregation. There is no update, no delete, so the design does not have to carry the machinery of in-place modification.
2. The speed of events — events arrive much, much faster than the time it takes to write a single transaction onto a table, so the design must absorb that speed.

Because we know we never perform update and delete on the table, and because we must mitigate the speed at which messages arrive, we can minimize and simplify the overall system design. These two aspects are the primary motivation behind the architectures of data intensive applications — and why they are different from traditional architectures. Different architectures are examined in detail in the sessions that follow.

2.6.6 Student Questions and Answers

Real-world & domain connection: This exact journey is documented in real systems — Marz's SuperWebAnalytics case shows the same arc (incremental counters → queue → sharding → corruption and fault-tolerance pain) as the motivation for the Lambda Architecture. Modern web analytics products (Google Analytics, Mixpanel) and CDNs' hit counters face the same write storm; their solution is the big data shape — append event logs, aggregate in batch and streaming layers — precisely the architecture family this session motivates. Netflix, LinkedIn, and Uber run variations of these systems on Kafka + Spark/Flink.

2.7 The Data Model: Atomicity, Immutability, Eternity

2.7.1 The Three Properties

The data model for these systems has three considerations: atomicity, immutability, and eternity. Eternity here means purity. Each property constrains how events are written, kept, and read:

2.7.2 Atomicity and Fine-Grained Data

Atomicity: when you perform a transaction, either the transaction is complete or it is not done — there is nothing like an intermediate state. Because each transaction is a complete, indivisible unit, the data is fine-grained. And the payoff is easy to see: we want data to be fine-grained because querying will be efficient.

2.7.3 Immutability

Immutability: you don't want to change the data. The original data is immutable. The only transactions that are possible are the two from section 2.1 — filtering and aggregation — and because those never modify the source, the data stays immutable. No update, no delete, ever.

2.7.4 Eternity and Purity

Eternity: the data is always pure. How is this achieved on message data? Two mechanisms. First, each message is associated with a timestamp — every event is anchored in time. Second, purity also comes from immutability: because the data is immutable and never modified, the data is always pure. There is no way for the stored events to drift from what actually happened.

Eternity means purity: a record that can be rewritten is a record whose truth is at the mercy of every later edit. A timestamped, immutable event has no such vulnerability — it cannot drift from what actually happened. Purity is not a mood; it is the joint guarantee of "anchored in time" + "never modified."

2.7.5 The Two Primary Motivations, Restated

Put together: the nature of the data and the kind of operations we do on it (only filtering and aggregation), and the management of speed at which events arrive (much faster than writing a single transaction to a table), are the two aspects that drive everything. We know we don't perform update and delete operations on this table, so the overall system design can be simplified — while still mitigating the speed at which messages come in. That simplification is the primary motivation behind the architectures of data intensive applications.

Real-world & domain connection: These three properties are exactly what modern event-sourcing systems implement. Kafka's log is immutable and append-only; each record carries a timestamp, and consumers replay the log — purity by construction. Event-sourced banking systems rebuild balances from immutable transaction events instead of storing a mutable balance (the friend-count vs. friend-list distinction in the reference literature is the same idea). Stream analytics on AWS Kinesis and Google Pub/Sub store events the same way, and the "eternally true, timestamped fact" model is the foundation of the Lambda Architecture's master dataset.

2.8 Traditional ER Modeling vs the Big Data Data Model

2.8.1 The Traditional Entity–Relationship Model

In the traditional data model we use entity–relationship (ER) modeling. You declare an entity employee, a relation called works for, and you say: employee works for department. Then you attach attributes: the employee has an employee ID (the primary key), a name, a salary, and so on; the department has a department ID, a department name, and the number of people in it.

2.8.2 The Node Graph Model

The big data data model differs in one simple thing: here, the notion of a fixed schema is replaced — each instance is an entity, and you capture these entities in what is called a node graph. Take person as an entity: a person has a name, a date of birth, relationships with other persons, an age, a location. Now take an individual instance, person P1: P1 is related to person P2, and that relationship carries its own attributes — age, name, gender. Another relationship: P1 and P2 work at the same location — the relation is "same location." Another: a male with a particular name and a female with a particular name are proposing to each other for marriage — the relation itself is the content.

In the traditional system all of this was individual rows: these are all individual rows. In the big data model, each row becomes an entity, and every node in the graph can carry different attributes — the relations are defined record to record, not once at the schema level.

2.8.3 Instances as Messages, Not Fixed-Schema Rows

When we say each instance is an entity, we do not store the instances as a single table with a fixed schema — because these are messages now. Say a person named Ram and a person named Sita are sending text messages to each other; you want to process and analyze these events. Each single instance is a message. If you start loading those messages using the traditional model, the messages become rows and you end up with an entire table — and that is a bottleneck, because the write operations may not be faster.

2.8.4 Persist the Inference, Discard the Events

So how do big data systems process them? You don't store them at the fundamental place. You process the events, analyze them, and then discard the events — but you persist the analyzed artifact: the inference. As a single statement, "P1 and P2 are proposing each other" — that inference is persisted. Persisting that kind of inference is not the objective of the traditional data model.

2.8.5 Student Questions and Answers

Real-world & domain connection: This is the exact divide between relational modeling and graph databases (Neo4j, Amazon Neptune) and between schema-on-write and schema-on-read analytics. Real fraud engines do what the banking example describes: they analyze transaction events as they stream in, emit a fraud score, and act — while a data warehouse (the "other system") keeps the full transaction record for statements. LinkedIn's real-time feeds and social graph, which are node graphs over event streams, run the same way.

2.9 Complex Event Processing

2.9.1 What Complex Event Processing Is

There is a special case of event processing called complex event processing — CEP. In CEP, you look at all the snapshots of the streaming data and try to come up with some kind of inference on what these messages, what these snapshots, are talking about. The message stream is examined as a whole rather than event by event.

Even in CEP the messages are not voluminous: maybe it is not just KB — it may be at most around 1 MB — but the objective of creating the messages should be very clear to you as a data engineer. The design discipline from section 2.2 applies: you know you need to develop a message processing system; the first point is to define your message — what the message is in this use case, what content it should have, and how you extract that content and the numeric information from the ongoing context.

2.9.2 CEP in Practice

The historical question from the data model discussion is CEP territory: did person P1 propose to somebody else before, across the timeline? That kind of pattern analysis over the history of events is exactly what complex event processing does. Real-world: fraud analysis that examines not one transaction but a customer's sequence of transactions, and offer planning that looks across an entire season of events, are CEP-style applications.

Real-world & domain connection: CEP is a production discipline, not a theory curiosity. Fraud engines (Visa, Mastercard, bank risk platforms such as FICO Falcon) run sequence-pattern detection over transaction streams; market surveillance systems detect insider-trading patterns across order events; logistics systems infer shipment anomalies from tracking-event sequences. Dedicated CEP engines (Apache Flink CEP, Esper, Drools) implement exactly this whole-stream pattern analysis, and the "did P1 propose before?" question maps to the temporal-pattern queries these engines run.

2.10 The Architect Mindset and What Comes Next

2.10.1 Roles and Mindset

A closing note on how to approach this course: it is not very theory-based — it is a little more involved than that, because the focus is to develop an architect mindset. The kind of roles these concepts point to are data engineer and data architect, and this course is a transition toward those roles. The session closed with the reminder that the notes for a session may look rough, but the understanding of the subject is what matters.

2.10.2 What Comes Next: Architectures

The instructor announced that the architectural models begin from here — this session's motivation is the foundation — and the course moves to architectures in the coming sessions, with an assignment following in later sessions once enough topics are covered. After a few more sessions, once enough topics are covered, an assignment will follow in which you study some architectures based on the lines discussed in class.

Real-world & domain connection: The architect mindset is the day-to-day job of data engineers and data architects: defining message contracts with business teams, choosing retention and replication policies, judging when a system stops scaling, and designing event-driven architectures. This session is the first step of that transition — the assignments and architecture studies in the sessions ahead are the field exercises.

Key Industry Applications

• Banking fraud alerts — real-time credit card transaction monitoring: each transaction is an event; the system filters for events that look fraudulent and alerts the customer. This is filtering in action (section 2.1.2), and when the fraud decision needs the customer's whole transaction sequence rather than one event, the same pipeline becomes complex event processing (section 2.9).
• Bank offer analytics (credit card offers on Amazon) — the bank tracks all transactions from a particular merchant, measures transaction volume, and computes the share of transactions converted into EMIs, then redesigns offers for the next season (for example around Dasara). This is aggregation in action — many events collapsed into the few numbers the next season's budget decisions run on.
• E-commerce segmentation — purchases segmented by product type (electronics versus everyday purchases, say a gas stove), by vendor, and by whether EMI conversion happened, from the same event stream. The same immutable events feed every slice; segmentation is just different filtering and aggregation shapes over one store.
• Mobile transaction alerts — the real-world picture of a lightweight event: KB-sized JSON messages carrying timestamp, amount, and location (who, where, when from section 2.2.2) — small enough to fit the in-memory windows that keep alerting fast.
• Website page-hit counters — the worked design journey from direct server-side updates, to a queue absorbing rate mismatch, to database shards for parallel writes, to the fault tolerance wall — the motivation for streaming architectures (section 2.6), and the exact arc documented in the reference literature on big data's origins.
• Complex event processing — inferences drawn across a whole stream of snapshots: historical analysis of a person's relationship events, or a customer's transaction history, for fraud and planning decisions. The stream is examined as a whole, never event by event.
• Message formats — events arrive as JSON, XML, AVRO, or plain TXT; format handling becomes concrete with PySpark later in the course, where schema and serialization choices actually matter.
• Roles — the concepts in this session build the mindset of a data engineer or data architect: defining the message, judging the requirements, recognizing the scaling wall, and designing the model.