Consistency, BASE, MongoDB, and the Big Data Analytics Life Cycle

# Consistency, BASE, MongoDB, and the Big Data Analytics Life Cycle

2.1 The CAP Theorem: Consistency, Availability, and Partition Tolerance

2.1.1 The Three Properties Defined

We start with a recap of the reliability and availability metrics from the previous discussion — MTTR (mean time to repair), MTTF (mean time to failure), and MTTDF (mean time to data failure) — and the four fault tolerance configurations: load balancing, hot standby, warm standby, and cold standby. Those configurations set up the idea that systems live with failures instead of trying to avoid them completely. The CAP theorem sits right in that world: it names the three properties every distributed system would like to have, and then shows you cannot have all three at once.

CAP stands for Consistency, Availability, and Partition tolerance. Each word has a precise meaning in a distributed system, and the meanings are stricter than everyday usage.

Consistency — a read of an item from any node results in the same data across multiple nodes. The data resides on multiple nodes; it may be partitioned (split across nodes) and it may be replicated (copied onto more than one node). Whenever replicas exist, every copy must agree. Picture three nodes \(N_1, N_2, N_3\) holding replicas of a variable \(x\). Reading \(x\) from \(N_1\), \(N_2\), or \(N_3\) must return the same value.

This can happen in any distributed system because multiple nodes service requests; if an update is not communicated to all the replicas, users end up seeing a different value at different nodes.

Availability — a read or write request that is issued is always acknowledged, with a success or failure message, within a reasonable time. If the operation succeeded, the node returns success; otherwise it returns failure. The system answers instead of hanging. Availability can also be framed as the fraction of the time the system is up and responding to requests. A request that takes far longer than expected usually means the system is not available.

Partition tolerance — the system keeps working even when nodes are cut off from each other by a network failure, which is called a network partition.

2.1.2 What Happens When a Node Is Cut Off

Take three peers — a, b, and c — connected over a network. Peer c becomes disconnected because of a network partition; a and b remain connected to each other. Peer c now has two options when a request arrives.

1. Respond immediately with a failure message, because it cannot stay in sync with a and b.
2. Wait for the network problem to be fixed, holding the request and answering only once it is repaired.

The second option can take a long time; the user request may time out, and then we say the system is not available.

2.1.3 Only Two of the Three

The CAP theorem states that a distributed data system running over a cluster can only provide two of the three properties — consistency, availability, and partition tolerance. The intersection of all three is never possible. You can build a CA system (consistency and availability), a CP system (consistency and partition tolerance), or an AP system (availability and partition tolerance), but never all three.

CA means the design cares more about consistency and availability. CP means consistency and partition tolerance. AP means availability and partition tolerance. In practice, network partitions are bound to happen, so when a partition occurs the system must decide whether it picks consistency or availability. That is why real distributed technologies are largely either CP systems or AP systems.

Visual intuition: the theorem is usually drawn as the CAP triangle — one circle for each property, with the three circles overlapping. The middle region where all three overlap is labelled "impossible": no real system sits there. Real systems sit in one of the three pairwise overlaps. The takeaway of the picture: the diagram has exactly three pairwise overlaps and never a triple overlap — matching the two-of-three claim.

With the scope clear, here are the traps that appear in exams:

2.1.4 Student Questions and Answers

Recap + bridge: CAP names three desirable properties and proves you can keep only two. The natural next question is how much consistency a system keeps — which is exactly what the five consistency levels describe next.

2.2 The Five Consistency Levels

2.2.1 The Ladder from Strict to Eventual

Consistency can be delivered at different strengths. There are five levels, and they form a ladder: strict, linearizable, sequential, causal, and eventual. Strict is the most stringent; each level below it is a little less stringent. The difference between levels shows up in which orderings of reads and writes are allowed when several processes operate on the same data.

The running example uses four processes. \(P_1\) writes \(x = 5\), \(P_2\) writes \(y = 10\), and \(P_3\) and \(P_4\) read the values. The initial value of \(x\) and \(y\) is zero on every node.

Visual intuition: the levels are usually drawn as four horizontal lanes, one per process, with time running left to right; writes are marked with a filled dot and reads with an open dot. The differences between levels appear only when a read's dot overlaps a write's dot in real time. The takeaway: all five levels look identical when operations do not overlap — the ladder's differences are visible only inside the overlap zones.

2.2.2 Strict Consistency

Strict consistency is the strongest model. It orders reads and writes by real time: once a write finishes, every subsequent read must see the new value, even if the read overlaps the write. A read that starts before a write ends and finishes after it must still return the updated value.

If \(P_3\) had read \(x\) while \(P_1\)'s write was still in flight, strict consistency would not allow \(P_3\) to see the old value 0 — the read must return the updated value. Strict consistency is the reference point: a schedule that is strict automatically satisfies every weaker level below it.

2.2.3 Linearizable Consistency

Linearizable consistency acknowledges that write requests take time to propagate to all the copies. It does not impose an ordering within the overlapping time periods of reads and writes; it only requires that every operation appears to take effect at a single point between its start and end. That point is the linearization point — after it, no read may return the old value.

Notice that when \(P_2\) writes \(y\) and \(P_4\) reads \(y\) at roughly the same time, the two operations involve a different variable than the \(x\) operations, so there is no conflict. Overlapping operations on different variables do not matter.

The difference from strict consistency: strict says an operation that started after another must see its result even when they overlap. Linearizable allows the overlapping read to see the old value. If two operations do not overlap at all, linearizable and strict behave exactly the same.

2.2.4 Sequential Consistency

Sequential consistency is weaker than linearizable. Each process receives the updates in some order, and every process must receive them in the same order — but the updates are allowed to be delayed between processes.

Why is it not linearizable either? Under linearizable consistency, as soon as a write ends the update should be available on the other nodes. Here the update to \(y\) is delayed at \(P_4\) — it arrives after the update to \(x\), even though the write of \(y\) finished earlier. Linearizable would reject that delay; sequential consistency tolerates it, as long as every process sees the same arrival order.

2.2.5 Causal Consistency

Causal consistency only forces an order on operations that are causally related; independent operations may arrive in any order at any node.

When \(x\) and \(y\) are updated independently — \(x\) was not read before \(y\) was written — there is no causal relation between them. They can be received in any order. A schedule that would fail sequential consistency can still pass causal consistency, provided the order differences are limited to causally independent operations.

2.2.6 Eventual Consistency

Eventual consistency is the most relaxed level. Nodes may receive updates in any order at any point in time. The promise is only this: when writes stop, the system will converge — eventually every node holds the same values, and the system becomes consistent. The moment every user sees the same value (say \(x = 5\)), the system is consistent again. NoSQL databases generally follow eventual consistency.

If a schedule fails causal consistency, it fails every stronger level too; the only level it can still satisfy is eventual.

The ladder in one view:

With the scope in mind, here are the traps that appear in exams:

2.2.7 Student Questions and Answers

Recap + bridge: the five levels trade strength for practicality — weaker models are easier to deliver at scale. The BASE model next makes exactly that trade explicitly, favouring availability over consistency.

2.3 The BASE Model

2.3.1 What BASE Stands For

BASE is a database design principle built on the CAP theorem. It stands for Basically Available, Soft state, and Eventual consistency. Where CAP says a system must choose, BASE states the choice openly: the system is available all the time, but may not be consistent all the time. BASE favors availability over consistency, the opposite of the ACID guarantees used by traditional transactional systems.

BASE versus ACID:

2.3.2 Basic Availability

The database will always acknowledge a client's request, either in the form of the requested data or a failure notification. The system is always available.

The database may be in an inconsistent state when the data is read, and a repeated request may return different results.

2.3.3 Soft State

Soft state means the state of the database changes even though no user has written to it between two reads.

At one point in time the user reads a stale copy of the data; after the repair, the actual copy. That is a soft state — different values at different nodes for the same variable, a condition that does not last long.

2.3.4 Eventual Consistency

Reads by different clients immediately following a write may not return consistent results. The database only attains consistency once the changes have been propagated to all the nodes. While the database is converging, it is in that soft state. As soon as the network partition is repaired, consistency prevails again.

A note on convergence: a system that has reached eventual consistency is said to have converged, or achieved replica convergence. When replicas conflict, convergence is helped by repair techniques: read repair (a correction triggered when a read reveals a discrepancy), write repair (a correction triggered when a write does), and asynchronous repair (correction outside the read and write paths).

2.3.5 Where BASE Fits

BASE is not useful for transactional systems where consistency is the concern and a high level of consistency is required. It is useful for heavy workloads, social media applications, and monitoring data for non-real-time analysis — the places where consistency is not of prime importance. All NoSQL big data systems follow the BASE model.

With the scope clear, here are the traps that appear in exams:

Real-world & domain connection: BASE is the operating principle of the NoSQL big data world. Cassandra states its adherence to the availability and partition tolerance properties of the CAP theorem openly; MongoDB's configurable consistency (next section) sits on the same spectrum; and social media platforms absorb millions of likes and comments per minute precisely because they accept soft state and eventual consistency instead of blocking on global agreement.

2.4 MongoDB: Configuring Consistency

2.4.1 MongoDB Basics

MongoDB is a very popular NoSQL database used in industry. It is a document-oriented database: data is stored in the form of documents, typically JSON documents — JSON files. An RDBMS has tables, and a table contains records \(r_1, r_2, \dots, r_n\). MongoDB has a collection instead of a table, and a collection contains multiple documents; each document is a kind of record. You can run SQL against tables to retrieve data, and MongoDB offers a similar set of functionalities: apply filters and retrieve documents of interest based on conditions.

RDBMS versus MongoDB:

The point that matters for this topic is the read and write choices MongoDB offers for a flexible consistency trade-off with scale, performance, and durability.

2.4.2 Topology: Primary, Secondaries, and Elections

A typical MongoDB deployment has multiple nodes: one primary and several secondaries. A client application typically interacts with the primary node — reads and writes arrive at the primary, and the primary decides how to perform them. Heartbeat messages pass between the nodes so each knows the others are alive. When the heartbeat messages stop, a node is considered failed. If the primary fails — or a network partition isolates it — a new primary must be elected, and after the election the operations resume. Once a new primary is elected, updates on it replicate to the secondaries.

The election, as a procedure:

1. Detect. The reachable nodes notice that the old primary's heartbeats have stopped.
2. Vote. The reachable nodes run an election among themselves; a node wins only if it can secure a majority of the votes.
3. Promote. The winning node becomes the new primary and resumes serving writes; the old primary is demoted.
4. Resync. Updates on the new primary replicate to the secondaries, and the old primary, once it reconnects, must catch up or be replaced.

Why a majority? The majority rule is what guarantees that two partitions cannot each elect a primary — only one side of a split can hold more than half the votes. That is the same "majority" idea that powers the consistency settings later in this section.

2.4.3 Choosing CP or AP

If every read and write goes through the primary, you are implementing a CP model: reads and writes happen through one node, so every read sees the updates from the latest write. Consistency is the priority; when a partition occurs, the system bothers more about consistency than availability.

MongoDB also allows reads to go to any secondary while writes still go through the primary. Now a read may not see the update from the latest write, because the update is still being communicated while the secondary serves the read. Consistency may suffer, but availability is high — while a write is happening, a read can still be served. That is the AP choice.

So the read path is the switch: CP if everything goes through the primary, AP if reads fan out to secondaries.

2.4.4 The Two Consistency Scenarios

Two cases show the trade-off in action. The setup is one primary and two secondaries, s1 and s2.

The wait is short, and eventually the system becomes consistent in both cases. You choose: for consistency, use the waiting option; for availability, use the immediate-answer option.

2.4.5 Read Concern

Read concern controls where the client reads from. There are three main options — local, available, and majority — and linearizable exists as a fourth option.

• local — the client reads from the primary replica.
• available — the read can be served from a secondary without causal guarantees: the client gets data that exists on the node it read from, which may lag behind the latest write.
• majority — the client only reads a value that the majority of nodes hold.
• linearizable — the strictest option: the read must reflect the latest acknowledged write, ordered by real time.

Majority is the best option for fault tolerance and durability.

2.4.6 Write Concern

Write concern controls when the system returns the acknowledgement for a write operation. The options:

• zero — not used in practice. The write has not even been applied; it sits in memory only.
• one — the write is acknowledged as soon as the primary is updated. This is the immediate-success path seen in the two cases above.
• n — acknowledge only once the write has been communicated to at least n nodes. For example, with \(n = 5\), success is returned only after five nodes received the update.
• majority — acknowledge once the majority of nodes have received the update, where majority means half plus one:

\[ \text{majority} = \left\lfloor \frac{n}{2} \right\rfloor + 1 \]

In words: take the number of nodes \(n\), halve it, round down, and add one. The lecture states the same rule as \(\frac{n}{2} + 1\) — "half plus one" — which gives the same result whenever the halving is done with whole numbers.

Two extra configuration knobs exist. A timeout bounds the write operation: once a write is issued, there is a limit on how long the system waits before declaring it failed. And the journal option controls durability: if journaling is true, nodes must save the write to disk before sending the acknowledgement. With journaling off, the acknowledgement is sent after writing to memory, which is faster but less durable; saving to disk is more durable.

2.4.7 The Four Consistency Scenarios

The timeline walkthrough has two processes: the old primary (p1) in one partition and a new primary (p2) elected in the other. The network has nine nodes; a partition splits it into a group of four and a group of five. In this setup:

Scenario 2 — causally consistent but not durable: read = majority, write = one. Writes are acknowledged as soon as the primary is updated, so writes are fast. Reads still wait for majority agreement, so reads are slower. A write accepted on the minority side may roll back later.

Scenario 3 — eventual consistency with durable writes: write = majority, read = local. Writes are durable, so \(w_1\) succeeds for p1, but reads are local — a read may not see the latest write, because the secondaries may not be updated yet. You get slow durable writes and fast non-causal reads. Real-world: a review site. People write long reviews and do not want them lost; the write must persist. Readers do not need a causal guarantee — the review does not have to appear immediately for every user; it appears eventually.

Scenario 4 — eventual consistency with no durability: read = local, write = one. A write is accepted as soon as one node accepts it, and reads always come from the primary, as in the previous case. Writes are not durable and may be rolled back — until a write is communicated, it can time out and roll back. Real-world: real-time sensor data feeds. They need fast writes to keep up with the rate of data, and reads want as much recent real-time data as possible. Data may be dropped during failures; that is an acceptable trade.

The best combination for the classic trade-off is read = majority with write = majority: causally consistent and durable. You do not return success to the user until \(\lfloor n/2 \rfloor + 1\) nodes have accepted the change, and you do not return a read result until \(\lfloor n/2 \rfloor + 1\) nodes have the same value.

2.4.8 Cassandra: Quorum Options

Cassandra offers similar options. Quorum is one of them, and quorum means majority — you want to achieve a similar thing to MongoDB's majority. Local quorum is also available: it takes care of consistency within one particular data center — the quorum is counted among the replica nodes in the same data center as the coordinator node, avoiding the latency of cross-data-center communication. There are also options to set the value of n directly, like the n write concern in MongoDB: ONE, TWO, and THREE let a write be acknowledged after one, two, or three replica nodes have recorded it.

With the scope in mind, here are the traps that appear in exams:

2.4.9 Student Questions and Answers

The second question in class probed the same waiting logic from the secondary's side — how a node knows when to give up on a message:

Recap + bridge: MongoDB turns the CAP trade-off into two dials, read concern and write concern. The next section leaves databases behind and walks through the end-to-end process that puts big data to work — the big data analytics life cycle.

2.5 The Big Data Analytics Life Cycle

2.5.1 Big Data versus Traditional Business Intelligence

To see why a life cycle is needed, compare big data with traditional business intelligence (BI). In a traditional BI environment, data is housed in a centralized server. In a big data environment, data resides in a distributed file system, and scaling is easy. Traditional BI largely analyzes data offline; big data offers both options — in-memory operations and real-time analysis, plus offline mode and batch processing. Traditional BI deals mostly with structured data; big data deals with a variety of data: structured, semi-structured, and unstructured.

The big data definition itself turns on three of these: volume, velocity, and variety. These distinct requirements need a step-by-step method that organizes the activities and tasks involved in acquiring, processing, analyzing, and repurposing the data.

2.5.2 The Nine Stages at a Glance

The specific data analytics life cycle has nine stages: business case evaluation; data identification; acquisition and filtering; extraction; validation; aggregation and representation; analysis; visualization; and utilization of analysis results. Each stage organizes and manages the tasks and activities associated with the analysis of big data.

Visual intuition: the life cycle is drawn as a circle of nine boxes, one per stage, with an arrow from each box to the next and a loop back from the last box to the first. The loop marks the point of the cycle: analysis is not a one-shot run — results get utilized, new questions appear, and the cycle starts again.

2.5.3 Stage 1: Business Case Evaluation

Before anything else, confirm that the requirement is really a big data problem. For it to qualify, it has to fit at least one of the needs in the big data definition: volume, velocity, or variety. If you have to deal with a high volume of unstructured data, that is likely a big data use case. Then the work must begin with a very well-defined business case that presents a clear understanding of the justification, the motivation, and the goals of carrying out the analysis.

The business case must be created, assessed, and approved by the management before proceeding with the actual analysis; you should think it through thoroughly and get approval before hands-on work.

The evaluation also defines the KPIs — key performance indicators — that let a decision maker understand which business resources the analysis will use and which business challenges it will tackle. The KPIs are the assessment criteria for the whole effort.

2.5.4 Stage 2: Data Identification

Figure out the data sources for the analysis. A wider variety of data sources increases the probability of finding hidden patterns and correlations. But too much variety can also confuse, and you may end up with an overfitting problem.

The required data sets and sources can be internal or external to the enterprise. Internal data sets come from operational systems directly; a data mart holds the historical data of one particular business unit (a related term in the data warehouse world, scoped to one business unit). External data sets come from third parties: organizations that provide financial details, medical bodies that provide healthcare data, publicly available data sets, and surveys — for which you decide whom to target. Sometimes data must be scraped from websites: the data is embedded in blogs or other content-based sites, and automated tools extract it — web scraping tools such as BeautifulSoup, a popular Python library for that job. Where an API exists, such as for Twitter data, you can use the API instead of scraping.

2.5.5 Stage 3: Acquisition and Filtering

The acquired data may include data that is not required for the analysis, or corrupt data. Filtering removes the bad data: empty responses in surveys, junk text inputs, attributes that the analysis does not need — you use a subset of the collected data based on what the analysis requires. In many cases involving unstructured external data, most of the acquired data is irrelevant and can be discarded as part of the filtering process. Corrupt data includes records with missing values, nonsensical values, or invalid data types.

Persistence is decided here too. Data needs to be persisted once it is generated and enters the enterprise boundary, so there must be safe storage. For batch analytics, data is posted to disk prior to the analysis. For real-time analytics, the data is analyzed first and then persisted to disk — the typical case for real-time analytics using in-memory computations.

2.5.6 Stage 4: Extraction

Extraction is dedicated to extracting the data and transforming it into a format that the underlying big data solution can use for analysis.

The extent of extraction and transformation depends on the type of analytics you will use and the capabilities of the big data solution.

2.5.7 Stage 5: Validation and Cleansing

Some of the data may not be valid. You specify validation rules, mark data that does not conform to them as invalid, and remove it.

Big data solutions often receive redundant data across different data sets, and that redundancy can be exploited: interconnected data sets help assemble validation parameters and fill in missing values — a value missing in one data set may be found in another.

For batch processing, validation and cleansing are achieved by an offline process called ETL — extraction, transformation, and loading. For real-time analytics, more complex in-memory systems are needed: data must be validated and cleansed as it arrives from the source, in main memory, and then analyzed on the fly.

2.5.8 Stage 6: Aggregation and Representation

Data arrives from multiple sources, and it must be merged across data sets onto a common field.

This gets complicated because the data structure and semantics differ among sources. One source may call a field surname; another calls it last name — you must pick one uniform name before analysis. Dates may be written as dd/mm/yy in one source and mm/dd/yy in another — you must choose a common format. The merging is effort-intensive on large data sets: it needs complex logic that executes automatically to fix all such things. You also have to take care of future analytics requirements, aggregating and reconciling the data to the needs of analyses that may come later.

2.5.9 Stage 7: Analysis

This stage carries out the actual analysis, and it may be iterative: if the analysis is exploratory, it is repeated until an appropriate pattern or correlation is discovered. It may also be confirmatory.

Confirmatory analysis is what you do in statistics: you state a hypothesis, then accept or reject it by applying statistical functions or rules — for example, a t-test decides whether the hypothesis holds. In exploratory analysis nothing is assumed in advance; there are no predetermined assumptions. The data is explored thoroughly to develop an understanding of the cause of a phenomenon — for example, market basket analysis: which two products are being sold together, and why. That understanding lets a store plan the layout, run cross-selling, or run promotions. Which kind applies depends on the analytic result you are looking for: descriptive and predictive analysis on survey data to understand the market fit for a new product can be as simple as writing SQL on the data and creating charts, or as difficult as building models on the data for hypothesis testing and prediction.

2.5.10 Stage 8: Visualization

Analysis is of no use if the interpretation stays only in the hands of the business analyst. The results must be easily interpreted by other users and stakeholders, so a powerful visualization is very important.

Real-world: Tableau is one of the popular visualization tools; bar charts, pie charts, and scatter plots are the common charts. A powerful visualization lets a business leader look at the interesting patterns and take better business decisions.

2.5.11 Stage 9: Utilization of Analysis Results

Finally, leverage the analysis. Suppose the analysis revealed certain customer behavior, and a recommendation system is already in place: feed the customer behavior into the recommendation system, and it produces more accurate recommendations. The results can improve the business logic or the application logic, serve as input for enterprise systems, drive business process optimization, or trigger alerts — emails or SMS sent to customers so a corrective action can be taken if something goes wrong. The results can be used in any of these forms.

Recap + bridge: the life cycle organizes the whole analytics effort, from business case to utilization. The next topic moves from the process to the machinery that runs it: distributed programming and MapReduce.

2.6 Private Clouds: Recap

2.6.1 Cost and Scalability

A private cloud is as good as buying your own infrastructure. You buy customized infrastructure built to your requirements, and you pay a high one-time cost — whether you use it or not, you have already paid for it.

Scalability is a challenge: you need to rethink what can be done to scale the infrastructure, because you cannot simply rent more capacity. The infrastructure can be managed by the organization itself, or by a third party: if the organization has the expertise, manage in-house; otherwise hand it to a third party.

Many automobile industries use cloud applications without having that expertise, while the IT industry typically does have it.

2.6.2 Student Questions and Answers

Exam Guidance Summary

The exam pointers for this lecture, in one place:

• The CAP theorem is very, very, very important. Know the three properties, the two-of-three claim, and why a partition forces a choice between consistency and availability (CP versus AP).
• The five consistency levels — strict, linearizable, sequential, causal, and eventual — are the conceptual core of this topic. Expect to reason about the example schedules: which ordering each level allows, and which it rejects.
• The distinction between strict and linearizable (overlapping operations) and between sequential and causal (dependent versus independent operations) is where the levels differ.
• Read and write concerns in MongoDB (local, available, majority; zero, one, n, majority) configure the same consistency trade-off; majority means \(n/2 + 1\) nodes — with nine nodes, that is five.
• The nine stages of the big data analytics life cycle are a structured method question: business case evaluation, data identification, acquisition and filtering, extraction, validation, aggregation and representation, analysis, visualization, and utilization of analysis results.
• The next topic continues with distributed programming and MapReduce.

Key Industry Applications

Where the concepts of this lecture meet production systems:

• MongoDB — the popular NoSQL document database; its read and write concerns put the CAP trade-off into production configuration.
• Cassandra — a distributed data store with quorum and local quorum options for the same majority-based consistency; the local quorum keeps consistency within one data center.
• Twitter — a real example of non-durable writes: a post can disappear and reappear on refresh, and rolled-back posts must be reposted.
• Review sites — durable writes with eventually consistent reads, so a long review is never lost.
• Real-time sensor data feeds — fast writes with no durability; data may be dropped on failures.
• ETL — the offline extraction-transformation-loading process for batch validation and cleansing.
• Tableau — a popular tool for the visualization stage, with bar charts, pie charts, and scatter plots.
• BeautifulSoup — a Python web-scraping tool for acquiring data embedded in websites; Twitter-style APIs are the alternative when available.
• Market basket analysis — exploratory analysis that drives store layout, cross-selling, and promotions.
• Recommendation systems — the utilization stage feeds discovered customer behavior back into an existing recommender for better recommendations.
• NoSQL big data systems generally follow the BASE model: basically available, soft state, eventually consistent.