Big Data Systems: Distributed Storage and Distributed Processing

12.1 Metadata: The Backbone of the Data Pipeline

12.1.1 Recap: Experimentation and Why Metadata Matters

The session starts where the previous one ended: how we train machine learning models, how we split data, and why metadata governs everything that follows. The model-building workflow ran through several stages. We decide how to select the model. We do feature engineering. We tune hyperparameters. We augment data to get more training material, and we validate the model. On the data side, we talked about training data versus test data, how to split the data using the 80-20 rule, and cross-validation.

Cross-validation has variants, including leave-one-out cross-validation, where each single sample takes its turn as the validation set while everything else trains the model. The split can also be randomized and shuffled, so the same data set can produce different train-test splits each time.

Metadata is the key to the whole data pipeline. It is what makes discovery possible: you can find a data set, an experiment, or a model again. It helps you understand the relationships between data sets. It governs data access and permissions, so the right people can reach the right data and no one else. It specifies the technicality of the data — its format, its schema, how it was produced — and it carries statistical descriptions about the data, like counts, ranges, and distributions.

Working from the reference material, these roles split into two broad classes you will be asked to classify:

12.1.2 The Metadata Artifacts: Dataset, Feature, Label, Model, and Pipeline

Every artifact in an experiment needs a reference: what the artifact is and what version it is. The discussion walked through the main artifact classes one by one.

Feature metadata exists for every particular feature. Each column of the data set gets its own metadata describing what the feature means, its type, and how it behaves. You do not describe features once at the data-set level; you describe every feature individually. The reference treatment tracks a feature version definition (a reference to the code that reads the raw data and turns it into the feature), the responsible person or team, creation and update dates, and use restrictions. Feature restrictions can be subtle: a feature may be legal to use in one jurisdiction but illegal in another, so the metadata must carry the conditions.

Label metadata captures the labeling process: why each label was assigned the way it was, and what version of the labeling rules was used. When a label changes meaning, the version tells you which convention a given sample follows. The reference adds label set version (because labels get corrected or new labels get added, and an older model may need the older label set for a fair comparison), label source (the human, the licensed provider, or the algorithm that produced the label), and label confidence (an automated labeler may earn less trust than a trained human labeler).

Model metadata records everything about the trained model: what parameters were used for training, what parameters were used for evaluation, how the model was used, which data set version trained it, and what the weights were. These are all very important pieces of the record. A model management system built on this metadata can answer: what configuration and hyperparameters produced this model, who authored it, which features it uses, and which snapshot of weights is in production.

Pipeline metadata describes the pipeline itself — data about the data pipeline. It becomes especially important when you build pipelines with advanced packages such as TensorFlow, because those pipelines are directed acyclic graphs (DAGs) of steps. In fact, we saw that data drift, concept drift, and performance drift can appear as a model moves from development, through training and staging, and into production; pipeline metadata is what lets you compare behavior across those stages. Reference material notes that some training systems record this automatically — ML Metadata (MLMD) is included in TensorFlow Extended (TFX) and stores artifacts about training runs, which pipeline run produced them, and which binaries produced them.

12.1.3 Experimentation Best Practices and the Model Package

During experimentation, one best practice is to capture everything about the run. The list recorded in class covers both the what and the why of each item:

A proper model package bundles the artifacts together: the model version, the evaluation record, the experiment versions, and the original model creator (the author). It also tracks modification and maintenance history, the downstream data sets that consume the model, and the artifacts used by the pipeline. Without this packaging, a model that worked in training cannot be reproduced or trusted in production.

In industry, this is exactly what modern ML platforms productize — experiment tracking systems that log data versions, environments, and metrics the way this lecture describes doing manually. The manual list and the productized tool solve the same problem: an unlabeled model is a liability, a labeled one is an asset.

12.1.4 Metadata Store, Repository, or Registry?

A metadata system can be built as one system or as multiple systems. You can decide which fits your pipeline; each option has pros and cons. The three forms discussed were the metadata store, the metadata repository, and the metadata registry.

A metadata store is a centralized place — one store, like an app store such as the Apple iStore or the Play Store. Inside it sits a metadata dashboard and a metadata database, and you access the whole thing through that central point. The store lets you see when metadata was logged and when it was updated, display the metadata, compare one version against another version, organize it, filter it, and query it.

A metadata repository is the choice when you want something like a GitHub-style place — the code base and everything around it living in one version-controlled space you can browse and understand. If you mainly need to access artifacts later, a registry (like a file registry) is the lighter option. The professor's mental model: for a model you might want a metadata store that works like a Wikipedia or a big repository — you go there to select the metadata you need for this model.

The decision connects to how you structure the pipeline itself. You can go pipeline-first or model-first — feeding the pipeline as a first-class object, or feeding the model and the experiment as first-class objects. Whatever you choose, orchestration tools such as Kubeflow and TensorFlow (which we studied with orchestration earlier) can drive the flow.

12.2 Traditional Data Analytics vs Big Data Analytics

12.2.1 Why Traditional Pipelines Fail at Scale

Everything so far was a recap. Now the lecture moved to a new theme: big data systems, distributed systems, and scalable systems — and the difference between traditional data processing and feature engineering versus big data systems. When you have a small amount of data, with little variation, it is easy to handle: a query, a report, a dashboard, done. When data is huge, has lots of variations, and carries so much complexity, the traditional pipeline data processing does not work as you expect. That is why big data systems exist: they combine with data science, machine learning, and deep learning to provide a better data pipeline. The two pillars underneath are distributed storage and distributed processing — you will meet both of them in detail later in this session, and every big data system in the world rests on these two ideas.

The whole data management story — building databases, building pipelines, building repositories, building metadata — exists for one reason: analytics. You mine data, you build data lakes, you build data pipelines, to derive some insight, to get business intelligence, to improve the business, to cut cost, to improve a process, or to do something right. Big data, or distributed data, is an umbrella term, much like cloud computing is an umbrella term. The data that forces this scale is everywhere: Netflix, Amazon, Flipkart, YouTube, Facebook, Twitter, and social media in general generate huge amounts of data that must be organized, stored, and pre-processed — and that is exactly where the data pipeline comes into the picture.

The human genome is a powerful example: processing genome data used to take 10 years; today it can be done in about a week by combining big data systems with AI. That is a roughly 500-fold speedup from architecture, not from waiting. Traditional database management systems and Excel files are still there, but beyond them sits the big data layer with its masses of data — even government data: census data, geographical data, and catalogs of government services all need to be collected and organized.

12.2.2 The Nature of the Data

The scale problem is not about the data alone: suppose you pull customer data from one source, from one system — that is easy. The same customer data coming from 20 different sources is difficult to manage. That is big data analytics — not a switch but a continuum. As more sources join, a normal approach stops being viable. There is no single line where "small data" becomes "big data"; the pipeline simply degrades gradually as sources multiply, and at some point the traditional approach stops being viable. The same data, at 1 source or 20 sources, is a different problem.

12.2.3 The Structure of the Data

Second is the structure of data. Data analytics mostly handles structured data: it analyzes structured tables to answer complex business queries and find solutions to business challenges. Big data deals with unstructured and raw data, and the main aim of big data is to convert raw data into meaningful data. This is one of the famous V characteristics — variety, which gets its own treatment in section 12.3.3. Data is not just a SQL structured table; it may be unstructured, semi-structured, audio, video, or a mix of everything.

The pipeline implication: a structured table has a fixed schema, so a traditional query planner knows exactly what it is looking at. Unstructured data has no such promise, so the pipeline's first job is often to impose structure — parse the text, decode the video frames, tag the audio — before any analysis can run. That is why "converting raw data into meaningful data" is the stated aim of big data, not a side effect.

12.2.4 The Tools and Technologies

Third is the tooling. Traditional analytics uses simple queries and reporting tools: you run a query, you build a report, you work in Power BI or Tableau, no problem. But when you try to combine so many things and push toward automation or reporting, it becomes very difficult. Big data needs sophisticated technological tools — automation tools, parallel computing tools, distributed computing, distributed processing — to manage the volume.

12.2.5 The Types of Industry

Fourth is the type of industry. Data analytics serves IT industries, travel industries, and healthcare industries; it helps them create new developments by using historical data and analyzing past trends and patterns. Big data is used by banking industries, retail industries, and many more; it helps them take strategic business decisions. Both matter, but they answer different questions: past trends versus strategy under scale.

When to pick which: if the data fits comfortably in one system and the question is about past trends, traditional analytics is faster, simpler, and cheaper. The moment the data spreads across many sources, loses its schema, or outgrows one machine, move to big data analytics — the switch is not a style choice but a scale choice.

12.3 The Seven Vs of Big Data

12.3.1 Volume

Big data refers to massive, complex data sets that traditional data management systems cannot handle. Such data has to be properly collected, managed, and analyzed. It is more than gigabytes: it is terabytes, petabytes, even zettabytes. The first V, volume, is the size of the data. The Internet of Things (IoT) is generating a lot of data — a traffic light generates data, YouTube generates data, Facebook and every social media platform generate data, transaction records generate data. You cannot deal with all of that in one store.

Even the database vendors hit this wall. In the old days one Oracle data file had a limitation of 2 GB, then they raised it, and now they make data files in terabytes, so a single big file can be terabyte-sized. The scale reference points are worth memorizing: a gigabyte is about \(10^9\) bytes, a terabyte about \(10^{12}\), a petabyte about \(10^{15}\), and a zettabyte about \(10^{21}\) — the size of the whole internet in the mid-2010s. Volume is the reason "one store" stops being the answer.

12.3.2 Velocity

Velocity is the speed at which data appears and disappears. The professor showed it with the live class itself: questions and answers, and the actions visible on video — data appearing and disappearing at speed. Traffic data, stock data, transaction data, log data, server data, heat generator data, temperature sensor data — all of it comes at high speed, appears, disappears, and pours into the pipeline. With such rapid data you need to make quick decisions.

The distinction that matters: volume asks "how much is sitting there?" while velocity asks "how fast is it moving past you?" A sensor reading every millisecond generates small data per reading but enormous data per hour — that is velocity doing the damage. Decisions that wait for the whole day's data may be decisions taken too late, which is why velocity is the V that motivates streaming (section 12.11).

12.3.3 Variety

Variety is the different types of data: structured, unstructured, semi-structured; Excel files, CSV files, JSON files, stream data, IoT data, image data, video data. There is no strict schema. That is why MongoDB became so popular — a document database does not force one schema on everything.

The mental model: a relational table demands every row have the same columns — that is a strict schema. Big data arrives in every shape, and a system that refuses to accept a shape cannot even hold the data, let alone analyze it. Variety is the V that pushed the industry toward schema-flexible storage: NoSQL document stores, object storage for raw files, and lakes that accept anything and sort it out later.

12.3.4 Veracity

Veracity refers to the accuracy and reliability of data. Big data comes in such great quantities and from so many sources that it can contain noise or errors, which can lead to poor decision making. Veracity applies to normal systems too. This is why data engineering exists — the cleaning step of the pipeline. When you do data engineering, you clean the data; in big data the cleaning is very, very big cleaning, because there is so much data and some of it is incomplete and noisy.

So you must make sure data entering a big data system is as accurate as possible, and when it is not, you do something about it: correct it, remove the noise, normalize the data. This is the data quality dimension work we studied before — data should be as accurate as possible and as relevant as possible.

12.3.5 Value

Value is the relevance of the data — the real-world benefits an organization can get from big data. High volume, high velocity, high variety, and accuracy are all nice, but for what? Finally, any data system exists to give value: value to the organization, value to the process, value to the people, value to the society, value to the company. If it is not valuable, why do we need it?

Value is the question that keeps the other Vs honest. A system can nail volume, velocity, variety, and veracity and still be pointless if no one can name what the data is for. In industry the value test is concrete: does the data change a decision, cut a cost, or improve a process? If not, the whole pipeline is an expensive hobby.

12.3.6 Variability

Variability is about preserving the context of the data. A coffee-ordering example: a customer ordered a coffee, but the system did not capture what blend, what sugar level — so next day the recommendation system suggests a different coffee because it did not capture the context correctly. The dosa example works the same way: in the morning the customer ordered a ghee dosa, in the afternoon an onion dosa, at night an egg dosa. If the context is not carried properly, you cannot make the right decision. Variability covers different blends, the unpredictability of customer behavior, and extreme conditions; the question is how you capture enough so the data is processed correctly.

12.3.7 The Seventh V: Visualization

Two more Vs complete the list beyond volume, velocity, variety, veracity, and value. The professor named variability, and then said the data has to be visualized — a common convention, so the seventh V is visualization: big data is of little use until it is shown to humans in a form they can act on.

The point is a closing loop. The first six Vs describe the data and its handling; the seventh describes the output. A human cannot hold a petabyte in their head, but they can hold a chart. Visualization is the translation step that turns the analyzed result into a decision — a dashboard a manager can read, a chart an operator can watch for anomalies, a map a city planner can act on. It is also the step that pays for everything else: an insight that is never seen is an insight that never happened.

12.3.8 Veracity vs Variability: Do Not Confuse Them

The single most important caution of this section: do not confuse veracity with variability. Veracity is data being as accurate as possible — the noise and error problem. Variability is preserving the context — the coffee blend, the sugar level, the time of day the order happened. One asks "is the record true?", the other asks "did we keep what makes the record meaningful?"

12.4 Goals, Use Cases, and Challenges of Big Data Analytics

12.4.1 What Big Data Analytics Is For

Big data analytics exists for faster, real-time decision making — technology-enabled analytics. It uses AI and supports both online and offline processing. The process runs: collect the data, store the data, pre-process the data, analyze the data, and get the results. The main thing to manage is latency: with lots of data spread across different systems, you need to extract, transform, load, perform analytics, and produce knowledge. Every step in that chain has to happen at scale.

Latency is the time between an event in the world and the decision that responds to it. A query over a small table answers in milliseconds; the same question over a distributed pile can take minutes or hours unless the pipeline is designed for speed. That is why the lecture names latency, not storage, as the main thing to manage: storage you can buy, but latency is a property of the whole chain — how fast data is collected, stored, pre-processed, analyzed, and turned into results.

12.4.2 The Principle of Locality

A key principle is locality: process where the data is. Locality means you go and fetch the data where it resides — relevant data, time-wise and space-wise — and you move the code near to the data.

The concrete version: stock data arrives at a location, and you want to flag high-volatility stocks. So you put some code at a location nearby, do the processing there, and transfer only the result to the next system. Fetching raw data from far away costs time and creates latency issues; locality avoids them. The flow is: keep data where it naturally lands, push the computation to the data, and move only the small output, never the large input.

This is not a minor optimization — it is the reason distributed file systems scatter data across machines and then schedule work on the machines that hold the data. Section 12.5 and 12.6 will build the mechanism (blocks on nodes, map tasks on the nodes that own the blocks); locality is the principle underneath both. Reference material states the same rule: the standard approach with HDFS and MapReduce is to locate the data blocks that need scanning and push individual map jobs out to those blocks, so the scan and processing stay local.

12.4.3 Real-World Use Cases

12.4.4 What Big Data Is Not

12.4.5 Challenges and How to Address Them

The challenge list is real: data integrity issues, quality issues, privacy issues, security, and compliance. You must define your objectives precisely, based on the challenges — what you want to do with big data, how you deal with a massive amount of data, how you will store it. Infrastructure must exist and scale. You need proper people. A data integration policy and a quality assurance policy are key — this is the core. Regular pipeline infrastructure and regular storage infrastructure are not enough; you must go for distributed storage and distributed processing.

12.5 Distributed Storage with HDFS

12.5.1 From One File to Many Blocks

The old way: everything stored in one file, in one place. The big data way: break the file into multiple blocks. Whatever the input — customer data, log data, a video file — it is split into blocks. For simplicity, think of a file split into five blocks. Block sizes vary: 2 MB, 4 MB, 16 MB are all normal, and the sizes step through a kilobytes series — 2 KB, 4 KB, 8 KB, 16 KB — and so on.

Now the blocks are scattered across machines. A node (one computer in the cluster, whether a physical machine, a virtual machine, or an AWS instance) holds some of the blocks. Block 2 of the file lives on node A, another copy on node B, another on node D; block 1 sits on nodes B, C, and E. The same block appears on multiple nodes.

12.5.2 Replication and Fault Tolerance

Why copies? Because big data systems run on commodity hardware — cheap machines, not very expensive hardware. When you consolidate low-cost machines, they can go wrong: think of three laptops working as a cluster; any one can fail. The default Hadoop distributed file system uses three-fold replication at any given time: each block is stored three times.

This is what a distributed file system means — distributed storage. It is the fundamental principle of big data systems. HDFS (the Hadoop Distributed File System) is a big data file system, written in Java, based on Google's GFS (Google File System), and rooted in work at Yahoo and Google. It provides redundant storage for massive amounts of data. It is written once and optimized for large file reads. Files are split into blocks at load time, blocks are spread across many machines, and each block is replicated across multiple machines.

12.5.3 Worked Example: Splitting a File into Five Blocks

12.5.4 The Name Node and the Data Nodes

Who keeps track of where everything is? The name node. The name node keeps the metadata: which block of which file is stored where — which node, which block, which location. It also knows the topology: whether the block sits in rack one of data center one or rack two of data center two. When a client reads a file, the client first asks the name node, and the name node points to the exact location; the client then pulls the block from the nearest place. The name node is the metadata brain of the file system.

12.5.5 Student Questions and Answers

12.6 Distributed Processing with MapReduce

12.6.1 From Distributed Storage to Distributed Processing

Storage was the first half. The second half is distributed processing — parallel processing. Earlier we took a 1 MB, 2 MB, or 100 MB file, split it into blocks, and distributed the blocks across machines: that is distributed storage. Now we process the video, the customer records, the job. The job is split into multiple smaller pieces — one piece for sorting, one for finding, one for grouping, one for clustering.

Whatever the big data ecosystem — Hadoop, AWS, Google, Azure — they all support these two things: distributed storage and distributed/parallel processing. If you remember only two sentences from this lecture, remember those two pillars.

12.6.2 The MapReduce Pipeline: Split, Map, Shuffle, Reduce

MapReduce is a programming model that uses parallel processing to process and analyze large amounts of data.

So instead of processing the file on a single machine, the job is split into many small tasks, and the tasks execute on different nodes using different storage blocks. That is distributed processing: processing is distributed. The one-sentence definition used in class: split massive jobs into smaller pieces, send them across different clusters, and patiently gather the results back together — no drama, no data loss.

The payoff can be stated with numbers. A job that runs in 10 minutes on one machine can run in 1 minute when its work is spread across the cluster — a ten-fold speedup. Written as a ratio, with \(T_1\) the time on one machine and \(T_n\) the time on \(n\) machines:

\[ S = \frac{T_1}{T_n} = \frac{10 \text{ minutes}}{1 \text{ minute}} = 10 \]

where \(S\) is the speedup. This is the idealized number: it assumes the work divides perfectly and nothing extra is paid for coordination. In practice the shuffle step costs time and network bandwidth, so the real speedup is smaller than the ideal — but the principle stands: the same work, spread, finishes faster. Ten tasks on ten processors, those processors possibly in different places, is the lecture's standing picture of distributed processing.

12.6.3 Worked Example: Word Count

12.6.4 Worked Example: Counting Global Car Sales

12.6.5 Standards, Compliance, and Metadata Control

What about the real-world complication: every country has its own compliance rules? The answer is standards. You decide which data must be given and in what format; the company follows the standard. Define the structure of the file and which inputs you want — no prices, no locations if a security guideline says so. That is where the pipeline comes in: you have to do the prep. In the import and export into the big data system, you control what is given and what is not.

12.6.6 Student Questions and Answers

12.7 Parallel vs Distributed Computing

12.7.1 Two Ways to Use Many Computers

Big data systems rely on two pillars — distributed storage and distributed processing — but the underlying computing models come in two flavors.

Parallel computing: multiple processors work at the same time, using the same memory. Each processor runs a symmetric multiprocessing arrangement; the processors share one memory and one system.

Distributed computing: multiple computing resources are connected in a network, and computing tasks are distributed across them. Not only the data is distributed — reading from 10 machines instead of one — the compute is too. Each node has its own processor and its own memory; nodes communicate by passing messages, and each machine keeps its own coupling to its memory. Parallel systems are tightly coupled computing; distributed systems are loosely coupled computers. A parallel system is one single machine, a tightly coupled cluster of multiple processors with a unified architecture — that is called a shared memory architecture. A distributed system spans multiple independent computers, each node with its own independent memory, where nodes pass messages to one another.

There is a control node — the master node — that coordinates. Multiple computing resources sit in the network, tasks are distributed across them: one machine processes something, another machine processes something else, and whichever node you need the data from — the nearby node, by locality of reference — performs the work. That is where load balancing comes into the picture: the master hands each task to the node that is ready and near the data, so no single machine drowns while others idle. When you need more computing resources to improve processing capability, you can go distributed; parallel systems grow inside one machine, distributed systems grow across machines.

12.7.2 Clusters: Old Wine in a New Bottle

The idea itself is not new. Before big data systems, Oracle already had a cluster — Oracle RAC, the Real Application Cluster. Microsoft has clusters; SQL Server clusters, DB2 clusters, Informix clusters — all the database systems, and not only database systems, can be clustered. Clustering means grouping things together; when you group, you need a common information bus that carries the data between members, and the messaging has to be clear — that is where fiber optic cable matters, because speed matters inside a cluster.

For distributed computing you do not worry about that: one system may be slow, another fast, no problem — but then you have latency issues. Distributed systems are not claimed to be efficient without effort: there is latency, but you can still use and manage it if you follow locality of reference properly. In short: cluster computing and distributed computing are old wine in a new bottle.

12.7.3 Student Questions and Answers

When to pick which: use parallel computing where everything must share one view fast (the metadata brain); use distributed computing where the data itself is big and partitioned — which, in a big data system, is most of the cluster.

12.8 In-Memory Computing with Apache Spark

12.8.1 The Memory Hierarchy

Memory registers are the fastest devices. Then comes cache — faster — then L2, then L3, then main memory, then local storage, then remote storage. Remote storage is what distributed storage looks like: cloud systems with data distributed in multiple places. It takes time to reach. But if you process on the system itself, it is faster.

The hierarchy reads like a distance ladder, each rung slower and larger than the one above:

The world is moving toward quantum computing, GPU-accelerated systems, and high-intensity CPUs and memory, and in-memory computing is where that trend lands: the more work you can keep in the fast levels, the less time you pay on the slow ones.

12.8.2 Why In-Memory Computing

In-memory computing means computing on the system itself: you do not go to the network to fetch something; everything happens inside the machine. Data goes into the cache — control injection, automatic calculations — gets processed, and comes back.

Every node has its own memory, so you process local data for reference — go there, process it in that CPU itself, keep the data in the memory instead of constantly reading from disk. That lets analysts and data scientists ask questions again and again without waiting, and never wait for data to stop flowing. Data streams in, sits in memory, gets processed — reacting in a millisecond. It is like the class itself: a question, an answer, done — it is in your memory, no waiting for a round trip to disk.

This is where Apache Spark enters: Spark is the in-memory engine for big data. Apache Spark is an open source big data processing engine designed for large-scale data processing. It is also the in-memory engine of the Hadoop system, and it runs on Linux, Windows, or Apple systems.

12.8.3 PySpark and How the Engine Works

PySpark is the Python API for Apache Spark. The engine's own language is Scala — a scripting language much like Python — and Spark itself is written in Scala. Traditional Python and traditional machine learning code are different from PySpark: PySpark takes the power of parallel processing.

The way to understand the engine: you all know the JVM, the Java Virtual Machine — the virtual machine that compiles and runs Java code. Similarly, the Spark in-memory engine takes your Python instruction, splits the Python code into small, small pieces, and if you have 4 CPUs, all four CPUs work in parallel, using the memory, computing faster and giving the results back. The Spark shell lets you run this on a local Windows machine — setup is easy — and your Python code runs faster.

Real-world: the same parallelism powers industry systems: Amazon, Walmart, and Trivago all use PySpark with machine learning to process faster.

12.8.4 PySpark at a Glance

Comparing a regular Python machine learning program with PySpark:

With PySpark and machine learning, you can do faster processing. For heavy workloads you can also run MLlib — Spark's machine learning library — with your regression code and neural network code. If time permits, running one MLlib code example live is the planned demo: download Apache Spark on Windows, follow the instructions, and run the code locally.

12.9 The Hadoop Ecosystem

12.9.1 Hadoop: The First Big Data System

Hadoop is the first and most popular big data system — an open source framework that has been around for a long, long time. When data comes to HDFS, it is broken into blocks and distributed on different nodes; you can access HDFS quickly through APIs. Hadoop daemons — processes running in the background — run on commodity hardware, and Hadoop became powerful the same way Android and Linux became popular: free, open, and easy to get started with. Hadoop is the big data answer to that story.

12.9.2 The Layered Stack

The Hadoop ecosystem is a layered approach: once the core is up, everything else installs on top of it, layer by layer. The components, with the one-line definitions to carry away:

The details that make the stack come alive:

• The bottom layer is HDFS — the big elephant — which stores a large amount of data. It is the core: data is not in one machine, it is in many machines, in many blocks. (You can also create the same layout on one machine with four disks — that is called a pseudo cluster — but the purpose is to distribute.)
• YARN — Yet Another Resource Negotiator — is a resource manager and scheduler. In my machine there is this memory and this CPU; you wanted to finish this job; YARN does the negotiation. It is also a resource management layer: you can edit its Java code if you need to.
• ZooKeeper coordinates between distributed components and makes sure everything works fine — like a zookeeper checking that all the animals are safe, that water is available, no issues. It is a software agent running like a daemon, checking everything. Ambari is an open source management platform for the cluster.
• The critical component is MapReduce — the yellow components were called critical; Google, AWS, and Microsoft all worked on these two, distributed storage and distributed processing. You can implement distributed processing with Python, Java, .NET, or C — no problem.
• Hive and Drill are the data warehouse layer — a big data warehouse, relational database in big data. You can do the drilling and data mining on Hive.
• HBase is a NoSQL database like MongoDB — distributed, scalable, built on top of HDFS. Hadoop provides HBase and it is very easy to set up once HDFS and YARN are running.
• Pig is a scripting language: to run 20 jobs or 25 jobs, use Pig scripting for data processing and data pipelines — pulling data from a file, loading it, filtering it, all in Pig scripting.
• Mahout and Spark MLlib are the machine learning libraries: complete machine learning library packages work using Mahout or Spark. You can set up Mahout and run a Python program that works beautifully for big data.
• Kafka and Storm handle streaming: videos, audios, traffic data, continuous data — that is the velocity V again. Kafka is for messaging, Storm for real-time stream processing.
• Solr and Lucene handle searching and indexing — where data is and how to find it. Lucene is a Java library for full-text search: give it one big book and it can search all of it.
• Oozie is the scheduling software.
• Flume and Sqoop move data in: Flume for unstructured streaming data such as videos (and Kafka can connect with Flume), Sqoop for databases — Oracle, MySQL, Postgres — and CSV files, where you define your control and push what you want as per the standard. That is where data validation standards come in.

12.9.3 HDFS Architecture: Master-Slave

HDFS uses a director-worker architecture — also called master-slave. The cluster includes one name node, which is the director server. The name node tracks where the file is, where the data is, where the permissions are, where everything is located; it manages the file system namespace. Data nodes are the worker nodes: each data node manages block creation, block deletion, block replication, and read and write requests. Each data node separately stores HDFS data in the local file system — this machine's disk, that machine's disk, a cluster of ordinary computers. The daemons run in the background on commodity hardware, and Hadoop is powerful because of that.

12.9.4 Worked Example: A Word Count Job on a Real Cluster

12.9.5 Setting Up Your Own Cluster: Ambari and the Cloud

Real-world: Apache Ambari is a cloud-based management system for building your own cluster. Like AWS, you can subscribe, create a virtual machine, work with your own VM instance — EC2 instances — and provision a cluster. Follow the installation guide, it gives you a URL, you go there and start working; on Windows you may need to stop the firewall. You can create users and do everything through Ambari.

The path is: provision machines (EC2 or your own), install the stack through Ambari's guided flow, and the cluster appears with HDFS and YARN running and the dashboard showing node health. What the lecture calls "about two classes to explain" is the configuration surface — the XML files, the firewall rules, the node setup — which Ambari turns into clicks.

12.9.6 Student Questions and Answers: Choosing a System

12.9.7 Student Questions and Answers: When NOT to Use Open Source

12.9.8 The HDFS + MapReduce Architecture

Putting it together: the data flow architecture of the ecosystem has an export-import process at the bottom; both the metadata and the standard plus the export-import process help you control the pipeline. The architecture is master-slave end to end: name node (director, metadata, namespace) over data nodes (workers, block management, local disks), with YARN negotiating resources, ZooKeeper watching health, and MapReduce jobs split into maps, shuffled, sorted, merged, and reduced across the cluster.

12.10 Apache Spark Architecture

12.10.1 Spark Core: The In-Memory Engine

Apache Spark's ecosystem is similar to the Hadoop system, but its core engine is different: Spark Core is the engine that does large-scale parallel distributed processing — in your memory. The professor's analogy: right now you are listening to a class, and other things are happening in your brain at the same time — humans are capable of parallel tasks, storing and processing multiple things at once. Spark distributes memory blocks and distributed processing jobs within the same machine the same way.

Spark has a well-defined layered architecture: its components and layers are loosely coupled and further integrated — you can execute the Z-land distributed database, RDD, and the directed acyclic graph (DAG). Spark loads data directly into memory and speeds up algorithms; execution is distributed among processors — instead of one processor, many processors run in parallel.

12.10.2 Resilient Distributed Datasets (RDD)

The key concept is the Resilient Distributed Dataset (RDD). What does that mean? A data set — say a CSV file — is normally loaded into memory as one frame. Instead, the RDD splits that data set into pieces: 10,000 records are split and the pieces are given to different processors, one piece per process. The entire program is then distributed — being done parallelly. You control the operations through the driver, deciding which runs first and which next.

12.10.3 Worked Example: Splitting 10,000 Records

12.10.4 The Spark Libraries

On top of the core engine, more libraries handle each workload:

The libraries share the core engine, so a streaming job, an ML training job, and a SQL query all run on the same in-memory parallelism instead of each owning a separate system.

12.10.5 Driver, Cluster Manager, and Workers

12.11 Stream Processing

12.11.1 What Streaming Means

Streaming is data continuously flowing. Like steam coming from a cooker at home: a little steam, then more steam, continuously. Traffic data is stream data; YouTube streaming is stream data. When more data comes, processing runs continuously. Input arrives, and which level comes first is processed first — level one, level two — the pipeline adds it incrementally. It is not "collect all the streaming data and then process"; it is process as it comes. The techniques reduce latency and allow incremental processing.

The contrast to hold onto: batch processing waits for a full chunk (say a day's worth of data) and then runs; streaming processes each arrival as it lands. Reference material makes the same point: nearly all data is produced continually at its source, and batch is simply a convenient way of processing that stream in large chunks. Streaming trades away the wait and gets latency down to near real time — the data is available to a downstream system a short time after it is produced.

12.11.2 Worked Example: Processing a Three-Hour Movie

12.11.3 Streaming Frameworks

The frameworks for streaming — real-time and batch — include Apache Spark (via Spark Streaming), Apache Flink, Ray, and Dask. Flink is an open source distributed engine for stateful processing: you can save the state of what happened when the data came — the transaction log has come, take it into Flink, put it into the file system, and then do the processing. Ray is another architecture. On the ingestion side, every company has its own tools to push data: Kafka, Flume (seen earlier in the ecosystem), and Amazon Kinesis. With Spark you can also do structured streaming using a high-level API.

The key concepts across all of them: the resilient distributed data set, splitting the job, using Spark queries correctly, and Spark Streaming.

12.11.4 Student Questions and Answers

12.12 Big Data in the Cloud

12.12.1 Amazon's Big Data Stack

Real-world: Amazon gives every Hadoop component a cloud name. Instead of MapReduce, you use EMR — Elastic MapReduce; it is the same thing, just Amazon's name. EMRFS is Amazon's name for HDFS — the HDFS you studied is the same thing; you have the same blocks. The streaming component is Amazon Kinesis. Around it sit Redshift (the warehouse), Redshift Copy, Kafka, the DynamoDB Connector, Amazon DynamoDB itself, Amazon RDS, and Amazon Elasticsearch — the counterpart of the Solr/Lucene search layer. Whatever Hadoop component you know, AWS has a service with the same role.

The mapping is not cosmetic: an EMR job still splits the input, maps it, shuffles it, and reduces it — the four steps of section 12.6 run under the Amazon name. EMRFS still stores blocks with replication; the name node role is played by the EMR service layer. Your mental model from the Hadoop ecosystem transfers without retraining.

12.12.2 Google's Big Data Stack

Real-world: Google has the same story: Google Cloud Dataflow for the processing pipeline, BigQuery for querying big data (the Hive role), Bigtable as the wide-column NoSQL store (the HBase role), and Cloud Storage for the raw data (the HDFS role). Inherently it uses the same concepts as the Hadoop ecosystem — distributed storage and distributed processing, again.

12.12.3 The Same Two Concepts Everywhere

Whether it is Hadoop on your own hardware, AWS, or Google Cloud, every big data platform supports the same two things: distributed storage and distributed/parallel processing. Learn the concepts once — blocks, replication, name nodes, mappers, reducers, in-memory engines — and every vendor's stack is just a set of new names for the same architecture.

Exam Guidance Summary

Exam note: metadata is a high-value exam area for the comprehensive exam. Questions are still being drafted, but the expectation is clear:

• Be able to classify metadata: given an artifact, say which type it is — administrative metadata vs structural metadata — and what that metadata does for your pipeline.
• Know the artifact classes: dataset metadata (location, how collected, when created, restrictions, licenses), feature metadata, label metadata (why labeled that way, labeling version), model metadata (training parameters, evaluation parameters, data set version, weights), and pipeline metadata (DAGs, drift between development, training, staging, and production).
• Know the experimentation best practices: capturing data version, configuration (YAML), environment, CPUs, memory, Dockerfile, Makefile, code version, hyperparameters, training and hardware metrics, evaluation, and performance.
• Know the model package: model version, evaluation record, experiment versions, author, modifications, downstream data sets, pipeline artifacts.
• Be able to compare metadata store vs repository vs registry, and explain pipeline-first vs model-first.

For big data, expect the seven Vs (volume, velocity, variety, veracity, value, variability, visualization) and the two pillars of every big data system: distributed storage (HDFS: blocks, replication, name node, data nodes) and distributed processing (MapReduce: split, map, shuffle, reduce). The distinction between veracity and variability was explicitly flagged as a confusion point — know it: veracity is about the record being true, variability is about keeping the context that makes it meaningful.

Also be ready for the supporting contrasts this session built: traditional analytics versus big data analytics (nature, structure, tools, industry), parallel versus distributed computing (shared memory versus own memory, scale-up versus scale-out), batch versus streaming (process after collection versus process as it comes), and the vendor mappings (EMR/EMRFS/Kinesis on AWS; Dataflow/BigQuery/Bigtable/Cloud Storage on Google).

Full depth on HDFS, MapReduce, YARN, and the ecosystem will be revisited in the big data systems course next semester, so this session is the conceptual foundation, not the final word — the exam tests the concepts, and the next course tests the machinery.

Key Industry Applications

• Amazon — personalizes advertisements and shopping recommendations from purchase history; runs Alexa voice analysis, customer segmentation, and fraud detection on big data analytics.
• Ginger — monitors mental health symptoms from user data.
• Kia Motors — identifies patterns and anomalies, including water quality issues and car control data.
• Walmart — generates roughly one million customer transactions every hour, which demands big data processing for analytics.
• Facebook — historically ~40 billion photos, an example of volume that only distributed storage handles.
• Human genome processing — took ~10 years in the old days and can be done in about a week by combining big data systems with AI.
• Amazon, Walmart, Trivago — industry users of PySpark with machine learning for faster processing.
• Toyota / global auto sales — the lecture's MapReduce case study: counting Corolla, Camry, and electric car sales across ~300 countries on a 5-node cluster.
• AWS — EMR (MapReduce), EMRFS (HDFS), Kinesis (streaming), Redshift, DynamoDB, RDS, Elasticsearch; Google Cloud — Dataflow, BigQuery, Bigtable, Cloud Storage: vendor names for the same distributed storage and distributed processing concepts.
• Databricks vs Hadoop — a real procurement decision: managed cost vs open source cost, buy vs rent.