Big Data, NoSQL, IoT, and Machine Learning

An architect today works with data that outgrows any single machine. These notes cover big data and its scale. They also cover the three dimensions behind every big data choice — variety, volume, and velocity — the application landscape, Hadoop and MapReduce, and Spark for real-time work. Later topics: Internet of Things basics, NoSQL databases, machine learning, and the security tools that guard identity and transport.

The thread tying all of these together is a single architectural shift: for most of computing history you brought the data to the program, because the program was bigger than the data. Big data flips that. The data is now bigger than any machine, so the program has to travel to the data — split into pieces, scattered across clusters of ordinary computers, and merged again. Every tool in this lecture (HDFS, MapReduce, Spark, NoSQL stores) is a different answer to one question: how do you store, process, and protect data that no single machine can hold?

14.1 Big Data: Definition and Scale

14.1.1 What Makes Data "Big"

The standard definition, as recorded on Wikipedia, is worth memorizing. Big data is a collection of data sets so large and complex that it becomes difficult to process them using on-hand database management tools or traditional database applications. Notice what the definition does not say: it does not give a size in bytes. "Big" is defined by failure — you simply do not get anywhere with the tools you have. The problem is not limited to relational systems either. Even non-relational databases become very hard to manage once you cross a certain size.

Relational database management systems (RDBMS) earned their reputation through indexing (a separate lookup structure, like a book's index, that lets the database find rows without scanning all of them): index storage and index access made a real difference in the past. But the internet, social media, and large-volume e-commerce changed the picture. Data became so big that parsing through it in any way is unimaginable. Searching it is also unimaginable, because the indexes alone would be too voluminous to build and keep. An index works when it is small enough to consult quickly; at web scale the index itself becomes a big data problem.

Real-world: this is why the architect's first duty is awareness. You must know that technologies like MongoDB, Hadoop, and Spark exist, even before you know them in depth. Short case presentations on the strengths and capabilities of these technologies are part of this course. People working on AI/ML are invited to share remarks from their own work, as are people working on IoT — especially medical devices.

The roadmap for this material: we look at IoT briefly, then NoSQL databases, then machine learning. We start with big data itself: the Hadoop file system, MapReduce, the idea of real-time analysis, in-memory databases such as Spark, and NoSQL databases in general.

14.1.2 The Storage Unit Ladder

To reason about big data you need the unit ladder. Each rung is a thousand of the previous one. A petabyte (PB) is one thousand terabytes. Nobody thinks small anymore: "beyond peta" is where big data starts.

Here is the full ladder, with each step worth $10^3$ times the one before it:

Consider some anchors so these units stop being abstract. The NASA Earth science data archive sits between 40 PB and 250 PB — decades of satellite imagery of the whole planet. On the consumer side, normally available hard disks go up to 10 TB, though most of us use drives of up to 2 TB. With the advent of solid state devices, access speeds have become almost magical compared to spinning disks.

One way to feel the size of a zettabyte: shrink a petabyte to a single byte. You would still need a million of those bytes to make one zettabyte — one "mega petabyte". Very large data centers work at the order of zettabytes. Beyond that sits the ronnabyte. Nothing existing is counted at that level yet. This is the powers-of-ten scale that the world's data is climbing.

Sense-check: the ladder is consistent — $1\text{ZB}=10^6\text{PB}$, and NASA's largest archive (250 PB) is roughly a four-millionth of a zettabyte. Individual organizations live in the petabyte range; only the whole world's yearly output reaches zettabytes.

14.1.3 How Fast the World Generates Data

A zettabyte is roughly the total volume of data generated every year across the world — whether stored or not. You may store only part of it; the rest simply exists, briefly or forever.

The current statistics make the rate concrete. Every minute, about 175 new websites come up. That is about three every second:

$$175\text{websites per minute}\approx 3\text{websites per second}$$

The social platforms show the same story at larger scale. Facebook users share 2.45 billion pieces of content each day. Feel the unit: giga means a billion — after a thousand comes mega (a million), then giga (a billion) — so 2.45 billion pieces of content, not gigabytes. Twitter, which we now call X, generates 560 GB of content every day. If you had to parse through 560 GB of content, even reading it once takes time. YouTube users upload about 500 hours of new video every minute of the day. Back in 2011 the figure was 48 hours of video uploaded every minute — roughly a tenth of today's rate. Look at the growth in just thirteen years. Enormous amounts of data get uploaded in one form or another. All of it requires processing.

Visual intuition: plot "hours of video uploaded per minute" against time and you get a steeply rising curve — 8 hours/minute in 2007, 48 in 2011, 100 by 2013, 500 today. The curve never flattens; each plateau becomes the next ramp. That shape, not any single number, is what "big data" means.

14.1.4 A Field Story: Telephone-Call Analytics

Two technical terms in that answer deserve unpacking, because they name the two standard ways of splitting a table across machines:

• Vertical partitioning splits a table by columns: frequently used columns (phone numbers, tower IDs) live on one node while rarely used columns live elsewhere. Each partition still has the same rows.
• Hash partitioning splits a table by rows: a hash function applied to a key (say, the phone number) decides which node stores each row. Hashing spreads rows evenly, so no node becomes a hotspot.

The lesson generalizes. When data flows faster than you can model it, strategy for handling — filtering, partitioning, distributing — matters more than schema perfection.

14.1.5 The Human Genome Project

Decoding the human genome shows how fast big data processing has improved. The Human Genome Project started in 1990. It took until 2003 — thirteen years — to uncover just 92% of the genome. It was a huge project and a big deal at the time: a worldwide consortium sequencing three billion DNA base pairs with the computing power of the era.

Today, complete DNA sequencing takes eight to nine weeks. Thirteen years down to weeks is a speedup of more than two orders of magnitude, driven as much by better algorithms and cluster processing as by faster chips. And the use of DNA sequencing has moved to a different level altogether: DNA-level treatment and DNA-level understanding of patients are now real practice — choosing cancer therapy based on the mutations in a patient's tumor, for example.

14.1.6 Facebook and LinkedIn at Scale

Two more scale anchors. Facebook maintains data hives (large warehouse tables built on top of Hadoop-style storage) that contain about 300 petabytes of data each. LinkedIn processes and mines petabytes of user data to power its People You May Know feature. The system crawls the connection graph and works out who may be known to you and how far removed one person is from another — friend-of-friend chains two hops out, acquaintance chains three or more hops out; Facebook offers similar features. The amount of data processed to answer such queries is stunning — hundreds of terabytes are processed every day just to answer them.

14.2 Variety, Volume, and Velocity

Three dimensions frame every big data decision: variety, volume, and velocity — often called the three Vs. Match your depth of planning to how much of each dimension your system actually faces. A payroll system faces almost none of them; a city surveillance network faces all three at once. The Vs are a checklist for deciding how much architectural effort a data problem deserves.

14.2.1 Variety: Not All Data Fits a Table

The variety of data is the first shock. Not all data can be put into tables. You have to keep track of click streams, logs, text, web logs with site visitors, sensor data, legacy documents, stored documents, images, audio, and video. Object storage (a store that keeps each item as an opaque object with a label and metadata, instead of rows and columns) covers much of this.

The useful split is between structured data (fits a fixed table: names, amounts, dates) and unstructured data (does not: a photo, a free-text complaint, a call recording). Both kinds demand a place in your design. A everyday analogy: structured data is like a filing cabinet of printed forms — every form has the same fields in the same order. Unstructured data is the shoebox next to it: receipts, photographs, a cassette tape. The cabinet is easy to sort; the shoebox is where most of the real information lives, and you need a different strategy for it. The analogy breaks when the shoebox grows faster than the cabinet — which is exactly what happens on the web.

14.2.2 Volume and Retention Choices

Volume is the second dimension. How much data is being stored? How much has to be processed? How long do you need to keep which data? There is a lot of working-out for the architect here, because retention is a choice, not an accident.

Retention decisions have real consequences in both directions. Keep everything forever and storage costs grow without limit, and old data becomes a liability (regulators and attackers both care about data you forgot you had). Delete too early and you lose the history that analytics needs — fraud detection, for instance, compares today's transactions against years of past behavior. So the architect writes explicit rules: transaction records kept seven years, application logs kept ninety days, surveillance footage kept thirty days unless flagged.

14.2.3 Velocity: Data on the Move

Velocity is the third dimension: data is on the move. Some data you are supposed to remember; some you are not.

The human brain does this beautifully, and it is worth studying as a storage policy. The brain catalogs what to keep based on sensory perceptions and how the experience got stored. Some data you will not forget in a lifetime — your name, a childhood accident. Some you will forget tomorrow — what you ate for lunch. Some never gets recorded properly — it lives in short-term memory and vanishes within seconds, like a phone number someone reads out before you find a pen. Similar strategies have to be worked out for real data: decide what deserves permanent storage, what deserves brief storage, and what deserves none.

Velocity shows up everywhere: Facebook feeds, share market and gold market entries and exits, tracking the movement of people. In each case the value of the data decays quickly — yesterday's stock tick or yesterday's location still has uses, but the decision that mattered was made seconds after the data arrived.

14.2.4 Surveillance Video Storage Strategy

Surveillance video is the classic velocity problem. A single camera recording continuously at full detail produces terabytes per week; a city grid produces more than any budget can store. The quantity generated is so huge that you need an adaptive storage strategy rather than blanket recording. Several levers exist:

1. Reduce the sampling rate to reduce the data. Fewer frames per second means fewer images to keep — at the cost of missing fast events.
2. Divide the screen into components. Only parts of the screen with motion get stored, and they get stored at a higher flash rate — the sampling rate. Static regions are not stored at all. An empty corridor contributes nothing; a person walking through it triggers storage of just their region.
3. Choose the depth of storage: how many gigapixels do you store per frame? More detail may be required when things speed up — a license plate needs more pixels than an empty parking lot.
4. Adapt to motion: the moment rapid movement is detected, increase the sampling rate and increase the detail stored.

So you can break the screen itself into components and store the video as parts of the screen. Innovative schemes like this turn an impossible stream into a manageable one. Notice the pattern, because it recurs throughout this lecture: spend storage only where the information is.

14.2.5 Archiving Documents and Images

Archiving covers scanned copies of records and complete transactions. Many financial accounting systems now store images of documents. Earlier, a voucher was made and signed, and all supporting documents were physically attached to it — a paper clip holding the invoice to the voucher, filed in a cabinet. Today the supporting documents are usually online. You link them up, or store them with the data. Anybody viewing the voucher then sees the supporting documents alongside it. This raises the bar even for an ordinary company whose records are handwritten — the images have to be taken and linked to the transactions.

Archiving is variety and volume working together: each scanned document is unstructured data (variety), and decades of transactions produce millions of them (volume). The link between voucher image and transaction record is what keeps the archive usable — an image nobody can find is indistinguishable from a lost document.

14.3 Domain Applications of Big Data

Big data earns its place through applications: banking, finance, retail, sentiment analysis, medicine, weather, industrial control, and customer service. Each application below shows a different face of the same three dimensions from the previous topic — variety in what arrives, volume in how much accumulates, velocity in how fast it must be acted on.

14.3.1 Medicine, Weather, and Industrial Plants

Medication today rests on statistics. Cancer medicine in particular depends on ethnic background, living habits, family background, and analysis of genetic patterns. Understanding one cancer patient generates a lot of data that must be examined — imaging scans, genomic sequences, treatment histories, outcomes of similar patients worldwide.

Weather prediction serves fishermen and farmers. Weather sites run mechanisms for picking up weather information, including volunteers who put up equipment and feed readings over the net. The weather data is analyzed to produce predictions that help farmers plan, help fishermen stay safe, and save people on the shore from hurricanes and tsunamis. This is a lot of data moving continuously — thousands of stations reporting temperature, pressure, wind, and humidity every few minutes, which is velocity in its purest form.

A nuclear plant or chemical plant processes its complete process data electronically. Many of you have worked with data loggers. A company called National Instruments was famous for virtual controls wired to physical transducers (devices that convert a physical quantity — temperature, pressure, flow — into an electrical signal). You create your own panel on the screen and design the knobs and switches. You connect them to transducers and write logical controls. Based on inputs, channels open, equipment switches on or off, lights come up, or alarms go off. On top of that you can log the data. Once logged, decisions based on analytics over that log again amount to huge data: months of sensor readings become the input for predicting when a valve will fail.

14.3.2 Banking Fraud Detection: A Worked Process

For flavor, watch the film Catch Me If You Can, with Tom Hanks. It is apparently based on a true story — an autobiography by Frank Abagnale, a young check forger who passed as a pilot, a doctor, and a lawyer. People cannot even establish how much of it is truth and how much imagination. To catch people like that today, you must be faster and quicker than they are.

Today even the manual step is automated. Banks have methods of determining irregular transactions, and they place actual calls to customers asking them to confirm whether a transaction really happened. The takeaway: storing large volumes is not enough. You need methods of processing the data at very high speed.

14.3.3 How Google Grew Into a Big Data Company

When we first knew Google, we only knew it as a search engine. Companies start narrow. When we first knew Microsoft, we only knew it made something called PC-DOS, then MS-DOS, and then we learned it wrote a language called BASIC. When we first knew Apple, we thought it was a nice tool for people interested in music, video, or photography to process images. Everybody has a beginning.

Google's beginning was cataloging data. They crawled websites, cataloged them, and created several terabytes of data. On top of that store they created the concept called big data — a method of storing data which gives good access. The collected data let them enrich their queries. They correct spelling mistakes ("it looks like you might want to search for this"). They consider synonyms. They guess what you are typing before you finish typing it. Their algorithms rank pages by the number of occurrences, the number of searches, and how many websites carry similar information. That combination is how pages get searched out for you.

The architectural lesson hides in the sequence: the catalog came first, the intelligence came second. Every smart feature — spell correction, synonyms, autocomplete — is analysis running on top of a massive stored catalog. No catalog, no intelligence.

14.3.4 Bigtable: A Sparse Multi-Dimensional Map

The storage design behind this revolutionized the field: automatically ordering rows through indexes and physical storage, and stamping rows and columns. Google stored data by creating a sparse spatial map of multiple dimensions with timestamps, and that moved them forward. At a certain stage they opened it up for others to use.

"Sparse" means most cells are simply absent: of all (row, column, time) combinations, only a few hold values, and empty cells cost nothing. Picture a spreadsheet with a million rows and a million columns where only a handful of cells are filled — yet any filled cell can be found instantly because its position encodes its meaning.

Real-world: Bigtable's design — sparse, multi-dimensional, timestamped — is the pattern to remember when you meet wide-column stores later in this lecture; Cassandra and HBase are its open descendants.

14.3.5 Cell-Level Sharing in Google Sheets

If you use Google Sheets, try this: you can give rights to each individual cell.

This example matters architecturally because it shows authentication (named accounts) and authorization (per-cell rights) — the two security ideas we will formalize at the end of this lecture — delivering a working multi-user application.

14.3.6 Yahoo and the Roots of Hadoop

Yahoo was another giant. There was a generation that swore by Yahoo — check a Yahoo email ID today and you can often guess the person's generation. Before WhatsApp groups, saying "groups" meant Yahoo groups. Yahoo created the Hadoop Distributed File System, and there was a time when Microsoft was keen to take control of Yahoo precisely because of the technologies it possessed. The lesson: today's unfashionable brand may hold tomorrow's infrastructure — Hadoop, which we cover next, still runs much of the world's batch processing.

14.3.7 Sentiment Analysis, Bots, and Ethics

Big data finds heavy use in banking, finance, retail, and sentiment analysis. Sentiment analysis (automatically deciding whether text expresses positive or negative feeling) is a major analytics topic. It featured in the last US presidential election. Analysts went through social media to find trends and fathom behavior. Automated bots held conversations with real people without revealing they were bots, molding and transforming opinions. That is the misuse.

Then comes ethics. What is right and what is not? Does Facebook have the right to see certain kinds of information? Does Facebook or X have the right to share information with law enforcement? This debate will continue, because privacy and security are two sides of the same coin. The less privacy, the more security; the more security, the less privacy. You have to draw a line somewhere. With the volume of data potentially available, privacy could go for a toss without enforcement and regulation.

A dictator controlling a country would probably like to know how many breaths each citizen has taken. An open society says: I have rights, and you may curtail them only when they start infringing the rights of others. Deciding where that line sits needs the right thinkers, not just engineers.

14.3.8 Automation, Monitoring, and Forecasting

Customer service is largely getting automated: robotic bots reply to conversations, sometimes more polite than thou. Monitoring of industrial equipment runs on large data volumes too — generating alerts, escalating appropriately, taking corrective action.

Weather forecasting has gone to another level. Advanced engines use global patterns of temperature, pressure, and winds to determine future weather. Forecasts are accurate enough that you can virtually decide whether to carry an umbrella. They give probabilities, so you decide what percentage risk you are willing to take. About two weeks before this session, uncontrolled rain inundated the UAE and threw Dubai into turmoil. The good part: people knew in advance it would happen, and most who took the warning seriously did not have to suffer.

On top of Bigtable-style storage, various analytics run: time series analysis, marketing analysis, financial analysis. IoT is heavily powered by data, as are graph tables. Applications include Hive at Facebook and data visualization generally.

14.3.9 Data Sharing Economics and Consent

Data sources come with levels of openness. There are rights governing what you may use and what you may not. Users sign agreements about sharing data in certain categories, and you must comply: you cannot use data the user did not authorize. You may even commit to not storing data unless the user approves.

Consent connects directly to business models. If a user does not share data, the provider may say: we will not be able to suggest the right advertisements for you. Free services live on advertising revenue, so the deal is simple: the service is free, but you watch ads. Real-world: on some flights, watching 30 seconds of advertisement buys you internet access at 36,000 feet for a limited time. It is a give and take: you give attention, they give connectivity.

As long as sharing is lawful on both sides, and advertisers agree to show ads only to people willing to see them, the exchange holds together. Some providers take money instead: pay a subscription up front and get the service without advertisements. Others let you select the type of advertisements you see — and for that selective targeting, the advertiser may pay a little more. All these models coexist.

14.4 Hadoop and MapReduce

14.4.1 The Hadoop Framework

Hadoop is an open source framework from the Apache Foundation, capable of processing large volumes of heterogeneous data in a distributed fashion across clusters of computers. It provides infrastructure for reliable storage, analysis, and persistence. It automatically maintains backups, so you can rely on the storage layer.

"Heterogeneous" is the word to underline: the data need not be clean tables. Logs, click streams, images, and sensor dumps all land in the same store, which is exactly what the variety dimension demands.

14.4.2 Name Nodes, Data Nodes, and Heartbeats

The Hadoop Distributed File System (HDFS) manages the storage. Files are automatically split into large blocks (fixed-size chunks, typically 128 MB) and saved on data nodes, with a backup mechanism between data nodes. The default replication factor is three: every block exists on three different machines, so the loss of any one machine loses nothing.

Name nodes — and you can keep backup name nodes too — distribute data to the data nodes and maintain the metadata for them: how many blocks each file has, which data node holds each block, and where the replicas live. The cluster is built from multiple commodity servers holding distributed storage.

Data nodes keep data in blocks. For better performance, the physical memory of the system holds the metadata and name node control structures, which speeds lookups — asking "where is block X?" becomes a memory-speed question instead of a disk-speed one. Faster modern storage helps every processing function further.

Data nodes provide a heartbeat: a regular "I am alive" signal sent to the name node. If the name node does not receive a heartbeat from a data node within a couple of minutes, it declares that node dead and starts replicating its blocks onto other nodes — the data was never lost because replicas exist elsewhere. The design goals: distribute, replicate, fault tolerance, self-healing. If data nodes fail, alternatives take over automatically.

Visual intuition: draw the cluster as a tree. At the root sits the name node (metadata only — it stores no actual file data). Below it, racks of data nodes hold the blocks. Every few seconds a thin line of heartbeats flows up from every leaf; when one leaf goes silent, the root quietly orders copies of its blocks onto other leaves. No human intervenes.

14.4.3 The MapReduce Pattern

MapReduce is the processing mechanism that allows work at multiple locations. It solves a problem every distributed system faces: moving data is expensive, so instead of bringing the data to the code, send small units of code to where the data already sits.

Complexity and cost: work scales roughly linearly with data volume divided by node count — double the nodes, halve the wait, until coordination overhead takes over. The practical limits are the shuffle phase (network-heavy) and jobs that need many passes over the same data, because MapReduce writes intermediate results back to disk between stages. That disk round-trip is exactly the cost Spark attacks next.

14.4.4 Where the Pattern Came From

While Yahoo and Google put in a lot of the initial work, the idea is older than big data platforms. Even in small data centers and companies, people with limited equipment broke jobs into smaller components, ran them at multiple locations, and merged results. If a single computer lacked capacity, you split the data, processed it in batches on the same computer, and merged the batch results. As machines became cheaper — limited in capability, but comparable to yesterday's big machines — jobs started running at various locations and merging into final output. As a strategy it existed for a long time; bringing it to mature, industrial strength was done in a big way by Hadoop. Reference diagrams of Hadoop users and topology accompany this material and are worth a leisurely look.

14.5 Spark and Real-Time Analysis

14.5.1 Why Memory Changes Everything

Spark, also from the Apache Foundation, exists for quicker processing. The key difference: if the volumes of data can be handled in memory, Spark uses memory. Everything is done in memory. This is in-memory cluster computing (keeping the working dataset in RAM across a cluster of machines instead of on disks).

It matters because real-time work needs quick processing. Typically, NewSQL databases with very large capacities that do MapReduce cannot handle real-time demands. Spark is the alternative created for real-time activity. Spark creates clusters and tasks across worker nodes; data is kept in cache and put together. If you have large volumes that must be processed online, this is the alternative to reach for.

A concrete comparison makes the speedup visible: a multi-step job that MapReduce runs by writing each intermediate result to disk might take minutes; Spark keeps those intermediates in RAM and finishes in seconds. For batch work done once a night, minutes are fine. For flagging a fraudulent card swipe while the customer waits at the counter, only seconds will do.

14.5.2 Real-Time Fraud Alerts

Real-time analysis in banking looks like this. User activities are analyzed as they stream. Patterns of behavior are checked, and alerts are raised on suspicious ones. SMS messages go out to customers, and security personnel are informed. Requests are processed based on the data being processed right now.

Notice how this completes the fraud story from topic 14.3: the twice-daily report was batch analytics; this is the same detection logic moved into memory so the alert fires during the transaction, not after dinner. A sports platform example accompanies the reference material for this topic — live score feeds processed as they arrive rather than after the match.

14.5.3 Student Questions: Can You Build Your Own Cache?

The professor's own war story settles why the hand-rolled route lost:

14.6 Internet of Things Fundamentals

14.6.1 What Makes a Device an IoT Device

At the fundamental level, an Internet of Things (IoT) device is a piece of hardware with maybe some software written into an EPROM (Erasable Programmable Read-Only Memory — a chip that keeps its program when power is off but can be rewritten with special equipment). It has an IP identity that is accessible remotely over IP. The communication channel can be anything: Bluetooth, Wi-Fi, or a GPS signal.

The device normally has an identity. It can access data from outside. Using sensors, it collects information from its local environment and transmits it as real-time information. The receiver is a controlling unit, which may sit in the cloud or anywhere remote. IoT devices are typically mobile: they go from place to place. A device may be non-mobile, but mobility is the common case.

So every IoT device is a velocity source in the three-V sense: it emits readings continuously, and each reading's value decays fast — a truck's location matters now, not tomorrow.

14.6.2 Sensors in the Field

Fixed installations exist too. Sensors fitted into soil measure moisture content or temperature and monitor plants regularly. Indoor cultivation is growing fast, and sensor arrays collect that data continuously. Earlier this was done with wired sensors, but wiring is not always viable — a greenhouse acreage or a field cannot be strung with cables economically. Wireless sensors now each carry an IP address, by which each can be located, and incoming data is analyzed as belonging to that area.

The addressing idea scales exactly like the internet itself: because every sensor has an identity, the receiving system can tell which field, which bed, which plant a reading came from — and can send commands back to a specific sensor, such as opening one irrigation valve.

14.6.3 Reference Architectures

A floor diagram of the Azure IoT reference architecture accompanies this material. Study it to see how identity, channels, and cloud control units fit together in a production design. In production designs the devices fall into patterns: some connect straight to cloud services over standard protocols; others reach the cloud through a local edge gateway that filters and pre-processes their messages first — useful where connectivity is poor, latency must be minimal, or raw telemetry should not leave the premises. Either way, the flow is the same shape: sense locally → transmit over a channel → process at a controlling unit → store and analyze as big data.

That last step closes the loop with this lecture: thousands of sensors each sending small readings produce exactly the continuous, high-velocity streams that Hadoop ingests in batch and Spark processes live.

14.7 NoSQL Databases

14.7.1 A Short History of Relational Databases

Most practitioners of an earlier generation started life with RDBMS, and even today it is comfortable to survive on them. Unify was popular once. Then Sybase rose, and Sybase and Oracle had a big contest going. People who wanted a free database used MySQL. Its syntax differed slightly from Oracle, particularly in where clauses. But it was freely available on the web, so anyone putting up a web page with a free database used MySQL. Later came MSSQL, which offered a free version for low-end usage called the Express Edition, and good bit of usage followed. Even today, leading database requirements at very large setups often stay on Oracle — continuity matters — while MSSQL gives everybody a run for their money. The products are now mature and dependable. The Azure cloud offering has made MSSQL a very strong contender.

This history explains why NoSQL had to fight for acceptance: relational products were not failing at ordinary scale — they were excellent. The pressure came from somewhere else.

14.7.2 Why NoSQL Emerged

NoSQL grew from a need for very large volumes of storage where the relational model was neither required nor practical. Web applications started generating so much data of varying types — audio, video, images, documents — that fixed tables stopped fitting. The concept of name-value pairs became popular. XML and other markup languages developed in parallel with this movement.

In an RDBMS you define columns up front and fill rows; any change in structure meant building new data structures — altering tables, migrating data, rewriting loads. NoSQL gives flexibility: add a new tag and it becomes a new column.

14.7.3 Name-Value Pairs and Flexible Rows

The storage unit is the pair: an address (the name) and the data (the value). Written out, you see the colon sign — on the left of the colon is what you are storing, on the right is the value:

{ "firstName": "Shane", "lastName": "Johnson" }

Decide to add a Twitter identity? Just add a tag: "twitter": followed by the handle. Curly braces indicate that the tags inside form a row. One row may have three columns, another two — perfectly in order. The metadata travels with the data, so the data never has to go elsewhere looking for its schema. It is a very nice structure — though you do not get the benefits of an RDBMS, as we see next.

An everyday analogy: an RDBMS row is a government form where every blank must be filled or marked "N/A" — change the form and every filed copy must be re-done. A NoSQL record is an index card in a recipe box: one card lists ingredients, another adds a note about wine pairing, nobody minds that the cards differ. The analogy breaks when you need to ask questions across all cards ("which recipes use saffron?") — the box has no index, and that is exactly the query weakness we meet below.

14.7.4 A Skills Database: NoSQL Versus SQL

{
  "userId": 1,
  "firstName": "Shane",
  "lastName": "Johnson",
  "skills": ["Java", "big data", "NoSQL"],
  "experience": [
    { "role": "product manager" },
    { "role": "technical manager" }
  ]
}

Put the same data in SQL and you need a fixed structure designed up front: a users table, a skills table with one row per skill per user, an experience table, foreign keys between them, and joins to reassemble what the NoJSON document shows in one glance. In NoSQL the structure is self-describing. If you know XML you will notice the resemblance: first name, last name, then skills as a set, then experience containing two items, each item carrying two labels.

The trade-off is querying. RDBMS gives you very good query capability; NoSQL does not — but NoSQL is very convenient to parse. When many parses run on different machines, having self-describing data is a genuine advantage. And when an RDB becomes unmanageable at scale, NoSQL stays flat: you just add storage and keep putting material in.

14.7.5 The Four Families of NoSQL Stores

NoSQL stores come in four types:

1. Document databases — keep the document as it is (JSON-style records like the skills example). MongoDB is the representative example; content management systems and e-commerce platforms thrive here.
2. Key-value stores — a name and a value, nothing more. The store cannot look inside the value; only the application knows what it means. A good example of key-value pairs in daily life is the Windows Registry — a name for each setting, a value behind it.
3. Column-oriented databases — store data by column rather than by row, so reading one attribute across millions of records touches only that column's files. These descend directly from Bigtable's sparse multi-dimensional map (topic 14.3.4); Cassandra and HBase are the standard examples.
4. Graph databases — entities interconnected by relationships (next subsection).

Take a quick look at MongoDB as the representative document store, and at Cassandra alongside it.

14.7.6 Consistency, Availability, and Partition Trade-offs

Here is the structural contrast. An RDB typically has a single node for reads and writes; an RDB cannot be partition tolerant. And within an RDB you face a blunt choice over indexes: fast reads need indexes, but the moment you add indexes, writes slow down — every insert must also update every index. So you choose between fast writes and fast reads by having indexes or not having them.

Consistency, availability, and partition tolerance form the triangle of compromise — formally the CAP theorem: when a network partition splits a distributed system, you must trade off consistency (every read sees the latest write) against availability (every request gets an answer). A NoSQL design lets you work on that compromise explicitly — many stores expose settings for how many replicas must acknowledge a read or write before it counts. In an RDB, you have it or you do not.

14.7.7 Graph Databases: A Worked Example

A graph database stores entities and the relationships between them. Build one from scratch:

• The Eiffel Tower is located in Paris.
• Paris is a city.
• The Louvre is in Paris.
• The Mona Lisa is exhibited at the Louvre.
• The Louvre is a museum.
• The Mona Lisa was created by Leonardo da Vinci.
• Bob is interested in the Mona Lisa.
• The Mona Lisa is a painting.
• Bob is a friend of Alice.
• Alice visited the Eiffel Tower.
• Bill also visited the Eiffel Tower.
• Alice is a person; so are Bob and Bill.
• The Eiffel Tower is a place.

Each sentence creates an entity, a relationship, or both. Entities become nodes (Eiffel Tower, Paris, Louvre, Mona Lisa, da Vinci, Bob, Alice, Bill); relationships become directed edges between them (located-in, exhibited-at, created-by, interested-in, friend-of, visited). You qualify relationships, direct them (one-way or bi-directional), and attach multiplicities.

Visual intuition: sketch the nodes as dots and the sentences as arrows. Two clusters appear immediately — a place/art cluster (Eiffel Tower–Paris–Louvre–Mona Lisa–da Vinci) and a people cluster (Alice–Bob–Bill) — joined by two bridges: Bob→interested-in→Mona Lisa and Alice/Bill→visited→Eiffel Tower. Those bridges are precisely what queries treasure: "what are Bob's friends interested in?" traverses two edges instead of scanning every person.

Such a knowledge graph is the cousin of the entity relationship (ER) diagram you draw for an RDBMS. Here, though, the graph itself is the database, not just a design artifact — and traversing persisted relationships is far faster than computing joins across tables at query time.

Certain patterns and partitions appear in the graph, and recognizing clusters becomes easy with practice. If you study discrete mathematics you learn to recognize these patterns, work with clusters, and understand why clusters matter when implementing databases. Graph theory is the underlying subject.

14.7.8 The Minimum RDBMS Kit for Architects

Even in a NoSQL week, do not skip relational basics. If you have been in the IT industry for a year or two without exposure to RDBMS, find a way to get exposed. For survival you should have an idea of normalization up to third order (third normal form — structuring tables so no fact is stored twice and every field depends only on the key). Know how you normalize, how you denormalize, and what the impact is. Know how data is accessed, have a little idea what indexes are, and know what a primary key and a secondary key are. Not necessarily for this course — for survival in the profession. An in-memory database rounds out the storage picture, and supplementary reference material covers extra essentials worth knowing.

14.8 From Management Science to Data Science and Machine Learning

14.8.1 Old Methods, New Names

Old wine in new bottles. There used to be a subject called management science and marketing. In it we used Markov chains, linear regression, curve fitting (converting exponential forms to linear ones to fit them), game theory, linear programming, and decision trees. Today we have come full circle and talk as if data science appeared suddenly. It did not. What we now call analytics is that same hard work, previously done with paper and pencil, now automated.

The historical motivation is worth one sentence per method: Markov chains modeled systems where only the present state predicts the future; regression found straight-line trends in noisy numbers; curve fitting bent exponentials into lines so they could be fitted; game theory handled strategic opponents; linear programming optimized scarce resources; decision trees mapped choices under uncertainty. Every one of these is now an algorithm with a package behind it. Embedded technologies wrote software into machines that performed activity automatically; combine that with data science and you help machines take decisions. Sometimes you can sit back and enjoy while the machine handles routine tasks — and some non-routine tasks too.

14.8.2 What Machine Learning Is

The agenda for machine learning (ML): applications, types of algorithms, steps in building a model, ML architecture, and popular tools. The tools part is partly designed for self-study.

Keep the three terms sorted: machine learning is the pattern-studying capability, data science is the practice of applying it to data problems, and artificial intelligence is what you get when that capability takes over tasks humans used to do.

14.8.3 Three Ways Machines Learn: Supervised, Unsupervised, Reinforced

Learning comes in three flavors, and a simple fruit basket explains all three.

All three are just algorithms — code you can already write. The differences are worth pinning down side by side:

14.8.4 Core Algorithm Families

Four families cover most introductory work:

• Regression — fitting data into a pattern (a line or curve through noisy points).
• Clustering — recognizing groups.
• Neural networks — interconnections between ideas.
• Decision trees — yes/no decisions, with or without probabilistic factors.

Decision trees deserve the probability treatment spelled out. Take the chance of each activity appearing, along with the cost of each. Multiply the probability with the cost and you get an expected value; whichever branch has the more attractive expected value wins the decision:

Supervised learning in practice: you help the machine decide, feeding it lots of data and using that data to build the tree.

The star example of unsupervised learning is the robotic vacuum cleaner. It goes around the house. Once it finds a table, it remembers there is a table there. Next time it will not bang against it — it bypasses. It carries a laser that sweeps the room and draws a map of the house. Its sensors strike obstacles, and it marks the map: table over here. That becomes the learning experience, and the machine improves on itself.

Clustering assumes everything appears in clusters, so a fault inside a cluster gets corrected by the algorithm itself. If it suddenly finds a dot on your cheek in a photo, it can remove it when you say beautify or clean up — the dot does not belong to any skin-tone cluster, so the algorithm treats it as noise and erases it.

Reinforcement was the robot recognizing the table, as above. Deep learning goes further: it models the way the brain works, with independent thought points. Like neural networks, deep learning systems share images, sounds, and videos and come to common decisions — sound matching, appearance matching, activity matching — and point out exceptions. The working feels almost human.

14.8.5 Language Processing: Assistants and Chatbots

Language processing powers the assistants you know: Alexa and Google Home. They recognize commands spoken in your language and organize them into a command language. They execute the command and play the appropriate music — from YouTube or wherever — on the machine you selected. They understand natural language almost like a human being. Chatbots work similarly. You write a statement, the bot studies it, and recognizes what you may be looking for. It then provides the correct option so you can take a decision.

14.8.6 Neural Network Structure

You can have an inner layer, a hidden layer, and an outer layer. Calculations are done by the individual boxes — the nodes — in each layer. Inputs enter the inner layer, each hidden-layer node combines its inputs with learned weights and passes the result forward, and the outer layer produces the answer. "Deep" learning simply means many hidden layers stacked between the inner and outer ones. Fraud detection systems listed among the applications are built from libraries implementing exactly this structure.

Visual intuition: picture three columns of circles connected left to right by lines, every circle in one column linked to every circle in the next. Each line carries a weight — a number learned from training data. Signal flows left to right; error flows back right to left during training, adjusting the weights.

14.8.7 Tools: Python and Java

Python has become the language of choice in the AI/ML community for three reasons: ease of programming, quality of the language, and ease of alteration. You do not compile every time; it runs line by line. Stuck at a particular line? Keep correcting that line, change the code, and execute again. Vast libraries available for AI/ML make it very attractive.

For a stabilized product going out as part of a machine, many people prefer to recode in Java. Java is structured and strong, with very good database connectivity, which makes it the choice for many products. Graphics work — detecting graphic patterns, retaining, rotating, adjusting, and matching images — calls for a graphics library. The audio names OpenGL, though the abilities described match OpenCV. The two are easy to confuse: OpenGL renders graphics (drawing and rotating shapes on screen), while OpenCV analyzes images (detecting, matching, and adjusting pictures). For image pattern-matching work, OpenCV is the standard tool; OpenGL remains the choice when the goal is rendering rather than recognition.

14.9 Security Technologies and Tools

14.9.1 Transport Layer Security

Transport Layer Security (TLS) has advanced enormously over the last ten years. It has become very important, particularly since workloads moved to clouds and everybody started working remotely. Any architect's checklist starts here.

TLS is the protocol that turns an open network into a private channel: it encrypts traffic between your browser and a server, and it certifies that the server is who it claims to be — the padlock in the address bar. When employees log in from cafés and homes instead of office networks, that encrypted channel is often the only thing standing between credentials and eavesdroppers, which is exactly why its importance grew with remote work.

14.9.2 Authentication Versus Authorization

Any site you visit faces two distinct concerns. Authentication answers: who am I? Authorization answers: am I allowed to go into that room? OpenID and Open Authorization (OAuth) are the standard methods for identifying yourself and your permissions.

That last line is exactly the role OpenID and OAuth play across websites: OpenID lets you prove identity once ("sign in with...") and reuse it; OAuth lets one site act on your behalf at another within granted limits — letting an application read your contacts without ever seeing your password. The Google Sheets mark sheet from topic 14.3.5 used both ideas at once: named accounts authenticated the faculty, cell-level rights authorized them.

14.9.3 LDAP and Identity Management

Companies set up LDAP (Lightweight Directory Access Protocol) servers to control authorization and authentication. Think of LDAP as the company's internal directory — a tree of users, groups, and permissions that applications query to ask "is this person who they claim to be, and may they do this?" Sometimes authentication and authorization run on separate servers; LDAP unifies them.

Beyond LDAP, Identity and Access Management (IAM) is an industry of its own. Companies in this business control your IDs so that one unique ID works across any application of the company — and can reach beyond it. Within countries, identity management is handled by national systems: Aadhaar, for example, gives each resident a unique identity.

The firewall completes the perimeter picture; that topic continues in the next session. Even those not yet exposed to it should treat security as an important area of self-study.

Exam Guidance Summary

The practical consequence: an open book rewards navigation, not memorization. Prepare a mental map of where each topic lives — the three Vs in the big data section, the CAP trade-off in NoSQL, the expected-value formula in machine learning — so that under time pressure you turn to the right page on the first try.

• Exam note: short case presentations are expected on MongoDB, Hadoop, and Spark — their strengths and the capabilities people should know. Practitioners of AI/ML and IoT (especially medical devices) should prepare remarks from their own work.
• Exam note: keep the three dimensions — variety, volume, velocity — ready with one concrete example each; surveillance video strategy is a strong velocity example.
• Exam note: know the index trade-off (fast reads versus fast writes) and the consistency-availability-partition compromise for NoSQL versus RDB.
• Study advice: normalization up to third normal form, primary and secondary keys, and basic index ideas are survival knowledge for every architect.
• Study advice: the machine learning tools section and the security technologies section are designed partly for self-study; go through the reference material supplied with both.
• Peer chat threads during sessions are educative — keep reading them.

Key Industry Applications

• Search and catalogs: Google crawled and cataloged the web. It invented its big data storage style (Bigtable: sparse, multi-dimensional, timestamped). It improved search with spelling correction, synonyms, autocomplete guesses, and occurrence-based ranking. The catalog-first, intelligence-second order is the reusable pattern.
• Social networks: Facebook hives hold about 300 PB each, and 2.45 billion pieces of content are shared daily. LinkedIn mines petabytes for People You May Know, processing hundreds of terabytes daily — graph traversal at industrial scale.
• Media: YouTube takes about 500 hours of video uploads every minute (48 hours/minute back in 2011); Twitter/X generates 560 GB of content daily.
• Science: the Human Genome Project needed 13 years (1990–2003) for 92%. Complete DNA sequencing now takes 8–9 weeks, enabling DNA-level treatment.
• Policing and telecom: continuous filtering, vertical and hash partitioning, and distributed queries over telephone-call metadata for movement tracking.
• Banking: twice-daily irregular-transaction reports built from filtration, frequency analysis, and historical rule comparison. Today it is automated, with confirmation calls to customers. Spark-class in-memory engines raise alerts in real time.
• Medicine: cancer treatment informed by ethnic background, living habits, family background, and genetic patterns.
• Weather: volunteer-fed sensor networks feed forecasts protecting farmers, fishermen, and coastal populations; the UAE floods showed warnings saving those who heeded them.
• Industry: National Instruments virtual panels wired to transducers control plants and log process data for analytics. Equipment monitoring raises alerts, escalates, and corrects.
• Cloud platforms: Azure IoT reference architecture for device identity and control. OCI and Amazon S3 serve as interchangeable object storage. Google Sheets offers cell-level rights with full version history.
• Assistants: Alexa and Google Home convert natural speech into command language and act on it; chatbots resolve intent and serve options.

Every row of this list is the same story in a different costume: heterogeneous data arrives continuously, gets stored once in scalable form, and earns its keep through layered analysis — the exact architecture this lecture built from HDFS to Spark to NoSQL to machine learning.