Database Recovery, Checkpointing, and the NoSQL Shift

17.1 Comprehensive Examination: Scope, Strategy, and Grading

17.1.1 The Syllabus Is the Handout

The last session opens with a simple and blunt statement about the comprehensive examination: whatever is in the handout is part of it. That "whatever" really means whatever — it includes the topics written in the handout, the topics covered in the live sessions, and the topics covered in the recorded sessions. There is no hidden division between "the handout part" and "the taught part". If a topic appears anywhere in the handout, it is examinable, even if the live sessions never touched it. This is why the session spends its first twenty minutes on preparation strategy rather than new material: everyone needs to plan the remaining days with an accurate map of the syllabus.

The topic list itself is a review of the whole course: normalization, the ER model, the basics of SQL, relational algebra, relational calculus, B-tree and B+ tree indexing, database security, access control, and recovery — the last one being the main technical content of this final session. On indexing, the advice is blunt: B-trees and B+ trees must already be practiced. An open-book examination does not forgive "we do not know anything about this" for something that is part of the course.

Exam note: everything in the handout is part of the comprehensive examination, including live and recorded session topics. B-tree and B+ tree indexing must be practiced, not read once.

17.1.2 What This Course Is Really Judged On

Before any topic details, the session states the benchmarks the course is judged against. The first is experiential learning: you must have designed a database, worked on a database, edited a database, and ideally integrated that database into an application. The assignment was the component that carried the experiential learning load — actually building something, not just reading about building something. The second benchmark is knowledge of the internals of a database management system. The point is made with a comparison: plenty of people can write a select statement and do basic CRUD operations. The difference between you and everyone else is whether you can sit with a client, understand their requirements, design a schema, create the indexes, let the application access the database concurrently, and make sure it is recoverable — all in a professional manner. Query proficiency is a third benchmark, and so is the ability to keep a database optimized in challenging situations. These are the "north poles" against which performance is judged.

Real-world: the description of a professional database person — requirements gathering, schema design, index creation, concurrency, recovery — is exactly the skill set a working database administrator or backend engineer is paid for. An interview question like "your application is slow, what do you check first?" is answered from the internals and indexing portion of this list; a production incident is resolved from the concurrency and recovery portion. The course is deliberately built so that every benchmark maps to a real professional duty.

17.1.3 Relative Grading and the Fifty Percent Average

Grading is relative, and the session is explicit about what that means. If everyone scores 190 and you score 89, that is not good for you; if everyone scores 50 and you score 60, that is good for you. Individual marks out of 30 or out of 100 do not define a result by themselves — the position relative to everyone else does. This relative system is also what allows questions that are challenging and thorough.

The second part of the grading picture is the average guideline. The heuristic passed to the evaluation team is that the average for the course should be near 50 percent. For a 100-mark paper, the relative average should sit close to 50. The same idea applies component by component: if an assignment average was very low, say an average of 5 out of 30, that is a signal about how that component was graded, not about you personally. The 50-percent average is a guideline for the whole course, and it separates the people who practiced and thought hard from the people who came without spending time. Nobody is labelled a failure or a success — the marks simply separate skill from its absence.

17.1.4 Q&A: What Does Self-Study Mean?

A student asks about the "additional topics" section of the handout: query optimization, the XML data model, database programming, distributed databases and client-server architecture, and recent trends in databases. The question is whether these are part of the course, or whether they should be read after the exam simply to be more complete as a database person.

17.1.5 Q&A: Open-Book Rules and the Bound-Notes Strategy

A student raises a practical crisis: the published open-book examination guidelines say that handwritten notes, loose sheets, and solved solutions must not be carried. But this student had prepared simplified, lucid notes — tips from the course written in plain language instead of complex textbook prose — and those are exactly the notes that would help in the exam hall. What is really allowed?

The strategy point is made sharper with a second example: the reason the exam team may have banned loose notes is probably not hostility — it may be that loose sheets can be exchanged during the examination. Hypothesize the reason behind a rule before attacking the rule. If printouts of slides, photocopies, and handwritten notes are all separately questioned, think from the inspector's perspective about why each was restricted, then align your request with the shared objective: the exam exists so students can demonstrate skill, and the rules exist to keep it fair. Business schools teach exactly this pattern: when a problem arrives, never walk in and ask "what should I do now?" — walk in with the consequence, the options, the evaluation, and the recommendation. It works for an exam notice, for a manager, for anyone whose only job is to say yes.

17.1.6 Q&A: Marks, SQL Evaluation, and Answer-Writing Strategy

Three student questions land together here, and each one sharpens a different part of the examination strategy.

The answer-writing method is then generalized into a repeatable exam procedure. Whatever the question — "is this schedule serializable?", "is it ACID?", "is it concurrent?" — first write one line saying what you understand the question to be asking. Then write the definition: "for me, concurrency means conflict serializability; for me, serializability means this schedule is equivalent to a serial schedule". Then apply it, state the consequences, and give the verdict. Even with a time constraint, that is two or three minutes of writing if you have practiced. The statement "I am assuming conflict serializability as my parameter for deciding whether this schedule is serializable" helps the evaluator see exactly what you expected, and helps you get partial marks when your conclusion differs from the expected one.

17.1.7 The Written Exam as a Thinking Exercise

The session closes this topic with a larger argument: the written examination is not just a memory test. Writing answers under time pressure trains you to comprehend a question, organize your thoughts, process the information, and present it as a structured argument — three, four, five, ten statements that align with your initial thought. A person who is formally trained in this way presents a logical case on any subject, and the person listening has no option but to give them a patient ear. The written exam, in other words, is a personality-building exercise disguised as an assessment. That is the deeper reason the answer-writing structure matters — it is the same structure you will use in front of any client or any manager.

17.2 Concurrency Control Recap: From ACID to Schedules

17.2.1 The Story So Far: Client-Server, Transactions, ACID

Before recovery, the session runs a careful recap of the concurrency story developed over the previous weeks, because recovery only makes sense on top of that foundation. The setting: an application running in client-server mode, with people accessing it from all over the globe. Many people access the same application, and the application has a database behind it. The claim is precise: the application behaves properly — follows the right properties — only when the database itself follows the ACID properties: atomicity, consistency, isolation, and durability.

Atomicity means either everything works or nothing works. Isolation means every user feels like they are working alone — other people's work should not impact their work. Durability and consistency must also hold.

Every user action arrives as a transaction — a set of program instructions, a program that should either work entirely or not work at all. With a single processor, transactions are processed one after another. If transactions were always executed fully one after the other — transaction one completes, then transaction two, then transaction three — the ACID properties would automatically hold. But real execution interleaves instructions from different transactions according to the processor's scheduling, and the exact interleaving is a schedule — the transactions working at a particular point in time. The question for any schedule is whether it will still follow the ACID properties. Two schedules that produce the same effect are equivalent, and a schedule is serializable when it is equivalent to some serial schedule. Serializability, not the raw interleaving, is the property that protects ACID.

17.2.2 Conflict Serializability and the Precedence Graph

The first test discussed is conflict serializability. A conflicting operation involves two different transactions operating on the same data item where at least one of them is a write. For example, if transaction $T_1$ reads item $x$ and transaction $T_2$ writes item $x$ — written $r_1(x)$ and $w_2(x)$ — those two operations conflict when their order changes the outcome.

A quick worked picture in the lecture itself: three transactions $T_1$, $T_2$, and $T_3$ interleave at different timestamps; suppose there is a conflicting operation from $T_1$ to $T_2$ and from $T_2$ back to $T_1$ — the arrow loop means a cycle, and the schedule fails the conflict serializability test. The example above realizes that picture with concrete reads and writes: the loop $T_1\to T_2\to T_1$ already fails the test on its own, and the third transaction simply chains another edge into the same cycle.

Visual intuition: picture the graph on paper — transaction nodes as dots, conflict arrows as arrows between them. The axes here are not data axes; the graph is a relation diagram. Walk along the arrows with a finger: if your finger returns to the dot it started from, you have walked a cycle and the schedule fails. Landmarks to notice: a node with only outgoing arrows is a candidate to run first in the equivalent serial order; a node with only incoming arrows is a candidate to run last. The one-sentence takeaway: a cycle is the only thing that can kill conflict serializability.

Real-world: the precedence graph used here is the same idea as the wait-for graph used in deadlock detection — one graph shape, two different questions asked of it. The wait-for graph draws an arrow from a transaction that waits for a lock held by another; a cycle there means deadlock, and the system kills one transaction. The precedence graph draws an arrow from a transaction whose operation conflicts with and precedes another's; a cycle here means non-serializability, and the schedule is rejected or repaired. Learn one graph shape and you can answer both "is there a deadlock?" and "is this schedule serializable?"

17.2.3 View Serializability and the Blind-Write Case

View serializability is the weaker test. For a schedule to be view-equivalent to a serial schedule, three conditions must hold for every data item. First, the initial read of each data item must be done by the same transaction in both schedules. Second, the final write of each data item must be done by the same transaction in both schedules. Third, intermediate reads — a read of a value written by a particular transaction — must come from the same writing transaction in both schedules. In short: for each data item, same initial reader, same final writer, same intermediate read-write pairing. A student illustrates the mechanics: if transaction 3 initially reads data item Q in one schedule, the candidate serial schedule must also have transaction 3 reading Q first; if transaction 5 does the final write of item P, the candidate must also finish with transaction 5 writing P.

The trouble with view serializability is practical: there is no single graph test. You must take the schedule and check it against candidate serial orders one option at a time, and with $n$ transactions the checking effort grows so fast that it becomes a very long job. There is a classic shortcut. A schedule that is conflict serializable is always view serializable. So when someone asks "is this schedule view serializable?", the fast route is to check conflict serializability first; if it passes, stop — it is view serializable too, and ACID is protected. View serializability becomes interesting precisely when conflict serializability fails, and that happens in the presence of blind writes — a transaction writes a data item without reading it first. A blind write is the one situation where a schedule can be view serializable without being conflict serializable, because the write-write conflicts that the precedence graph would flag do not exist as read-write conflicts. So the decision procedure: check conflict serializability first (easy); if it passes, done; if it fails, only then consider whether blind writes make a view-equivalent serial order possible — and be prepared for the time cost of ruling out all the candidate orders.

The checking effort deserves a precise note. With $n$ transactions, the candidate serial orders number $n!$ — for $n=5$ that is already 120 orders to try. The professor quotes the effort as growing on the order of $n^2$ ("almost taking order of n square time") — that is the informal claim from the lecture; the standard textbook result is stronger and worse: testing view serializability is, in general, an exponential (NP-complete) problem, which is precisely why no efficient graph test exists and why the concept is taught for understanding rather than used for verification. Keep both pictures in mind: the growth is fast enough that you never want to do it by hand beyond a couple of transactions.

Comparison — conflict serializability versus view serializability:

When to pick which: use the conflict test first in every exam question; fall back to the view conditions only when the precedence graph has a cycle and blind writes are present.

17.2.4 Runtime Control: Timestamp-Based and Lock-Based Mechanisms

The serializability tests assume you know the schedule in advance — a schedule where transactions execute at different known points in time. In real operation, transactions keep arriving while the processor works, and you cannot predict when the processor will context-switch and give the turn to someone else. So at runtime, the database must decide, operation by operation: should I allow this transaction to read or write now?

Two mechanisms were discussed. The first is lock-based control: each data item carries locks. A shared lock allows reads; an exclusive lock allows writes; and the two-phase locking discipline — including its strict and rigorous variants — decides when a transaction may acquire and release them. At runtime, based on the locks held on each data item, the database allows or blocks each operation. The second is timestamp-based control: every transaction carries a timestamp, and every data item carries a read timestamp and a write timestamp. Comparing the transaction's timestamp against the data item's timestamps decides whether a read or write goes through. That is the entire gist of concurrency control — how a database allows operations to proceed concurrently while keeping schedules serializable.

17.2.5 Q&A: Revising Locks and Timestamps

Before moving to recovery, a student asks to revise the lock-based and timestamp-based mechanisms once again. The response refuses to re-teach the whole topic and turns the question into a lesson about responsibility — and then gives the one question that makes both mechanisms make sense.

17.3 Recovery: Failures, Logs, and Undo-Redo

17.3.1 Why Recovery Exists: Failure Types

The transition to recovery is motivated the way every topic in this course is motivated: understand the need first, because if you understand the need you can replicate the solution when a similar challenge appears in your real life. The need is that a transaction can fail at any time, and there are many kinds of failure. The most frequent is a power failure: the power goes, and when it comes back the system works again. But deeper failures exist too. Every hardware component fails — just like every system, hardware has limitations. The hard disk can fail, the processor can fail, the network can fail. The "army" that stores the database — the main memory and the secondary storage — can fail at any point in time, and the system must be prepared for that.

The consequence of a failure lands on the ACID properties. A transaction can work and commit, with the committed operations sitting in main memory, and then the power fails or the memory fails — or even the secondary storage fails. In that position, the changes that were made are not persistent: the durability of the committed transaction is impacted. And when the system must undo work, the consistency portion is what is at risk — undoing some changes while leaving others breaks the before-and-after constraints of a consistent database.

17.3.2 The Volatile-Memory Problem and ACID Impact

The power failure case deserves a precise model, because it is the most common failure and the one the recovery design must handle carefully. The processor executes instructions from main memory. There is a pointer that records the last instruction already executed, so after a restart the processor knows to continue from the next instruction. But the changes made by the executed instructions were made in main memory — and main memory is volatile. The power goes, the power comes back, and the operations that were done in main memory are not persistent. The pointer tells you where execution stopped; it does not tell you what the database lost.

This is also the point where recovery and concurrency meet. Suppose transaction $T_1$ writes data item $A$, and transaction $T_2$ reads that item $A$ later in the schedule. If $T_2$ commits before $T_1$ and a failure happens, the system has a problem. Transaction $T_1$ must be rolled back — there is no option for $T_1$: everything it wrote is undone. But $T_2$ has already committed, having read a value that $T_1$ wrote and then rolled back. For $T_2$ there are two options, and both are bad. Allow the committed $T_2$ to stand, and the database is inconsistent — $T_2$'s result is built on data that no longer exists. Undo the committed $T_2$, and durability is violated — a committed transaction came back. The resolution is recoverability: whenever a transaction reads a data item written by another transaction, the reading transaction must commit after the writing transaction commits. Only recoverable schedules are safe, because then a failure never forces a committed transaction to be undone.

17.3.3 Recoverability: Who Must Commit Before Whom

Recoverability is familiar from the timestamp-based discussion, but recovery makes it a hard requirement rather than a design choice. In a recoverable schedule, if $T_2$ reads $A$ written by $T_1$, then $\text{commit}(T_2)$ can only appear after $\text{commit}(T_1)$ in the schedule. That single rule prevents the cascade of rollbacks described above — there is no way to end up with multiple committed transactions depending on an uncommitted writer. The rule matters at the moment of failure: when the system restarts, it must be able to recover each transaction independently, without creating inconsistencies between committed transactions.

The standard terminology sits on top of this rule. A strict schedule strengthens it: no transaction reads or writes an item written by another transaction until that writer has finished — committed or rolled back — so recovery can undo a failed writer simply by restoring the old values, and cascading rollbacks never occur. Cascading rollback is the domino effect the rule exists to prevent: $T_1$ fails, so $T_2$ (who read $T_1$'s data) rolls back, so $T_3$ (who read $T_2$'s data) rolls back, and so on — one failing transaction dragging a chain of others down with it. The reference texts stress what the lecture implies: practical recovery mechanisms are designed so that cascading rollback is never required — recoverable, and in practice strict, schedules are the norm.

17.3.4 The Log File and Stable Storage

Recovery needs a record of what happened, and that record lives in a log file on stable storage. Stable storage is the Hollywood version of a vault: in the analogy, a stable-storage situation survives the rain, the thunderstorm, the tsunami, and the earthquake — if the power goes or comes, if the tsunami comes or goes, if the hardware fails, the log is still there. Stable storage is achieved by replication: multiple copies of the log stored at different places, at different times. The cost of full replication is very high, which is why the log is special — it is a very small portion of the database, written in a stable format, always available even after failure. The recovery model is a fail-stop model: the system fails, the system restarts, and after restart the system works correctly again — at least well enough to read the log files.

The log format is simple and fixed. When a transaction $T_i$ starts, the log records $⟨T_i,\text{start}⟩$. When $T_i$ writes item $X$, the log records the transaction, the item, the old value, and the new value:

$$⟨T_i,X,v_{\text{old}},v_{\text{new}}⟩$$

When $T_i$ commits, the log records $⟨T_i,\text{commit}⟩$. Nothing more and nothing less: only operations relevant to the database are logged — every other detail of the application is left out.

Real-world: the same log-before-change idea — write the old and new value down before touching the database — is the foundation of journaling in file systems, versioned storage, and even financial accounting, where every change leaves an audit trail. A file system journals its metadata updates; a ledger journal records every entry before the books are updated; a database logs every write before it is durable. The same record — "item, old value, new value" — appears in all of them because the same need appears in all of them: if the machine dies mid-change, something must be able to tell the difference between "the change happened" and "the change did not happen."

17.3.5 The Worked Log Example

The example is built around a small social media application where members perform transactions at different points in time — say one member's transaction is logged as $T_1$, another's as $T_2$, and so on. As soon as anyone does anything, the change is written to the log in the fixed format.

17.3.6 Immediate Versus Deferred Modification

A design choice appears here: when the database itself is modified. With immediate modification, every change made in main memory is written back to secondary storage immediately. With deferred modification, the changes wait. The distinction matters because power failures are far more frequent than hardware failures, and random access memory is volatile: power goes, and changes that lived only in main memory vanish. In a power failure, the processor needs to know exactly what changes were done so it can continue from the right point — the log, with its old and new values, supplies exactly that knowledge. Immediate modification means the database on disk always tracks the most recent change, and the log decides afterward which of those changes are real (redo) and which are phantom (undo).

Comparison — immediate versus deferred modification:

The professor's emphasis is on immediate modification, and the reason is frequency: power failures are far more frequent than hardware failures, so the recovery design is tuned for the case where the disk holds recent-but-not-final changes. The log's old and new values are exactly the information the processor needs after restart: redo the changes that are real (committed), undo the changes that are phantom (uncommitted).

17.3.7 Undo, Redo, and Idempotency

The recovery operations themselves are familiar from everyday life. Undo means: whatever I did, I take it back — the way Ctrl-Z undoes the last change. Redo means: do it again — the way Ctrl-Y redoes a change in a word processor. The subtle and important point is why redo is safe. Redoing an operation must not change the result — the property the source describes with the garbled phrase "a.a is equal to": the property of idempotency — repeating an operation leaves the result unchanged. Concretely: if the log says $A$ changed from 1000 to 950, then writing $A=950$ once, and writing $A=950$ again, both leave $A$ at 950:

$$\text{write}(A,950)\Rightarrow A=950,\text{write}(A,950)\Rightarrow A=950$$

Because the redo operation is idempotent — repeat it twice and the outcome is identical — the recovery algorithm can apply a redone operation even if the database had already been written back before the failure. The database might already contain the value, and redoing it is harmless. Undo works the other way: undoing restores the old value, so for $A$ the undo would return 1000.

17.3.8 The Basic Recovery Algorithm

With the log understood, the recovery algorithm is a scan. Look at the log and classify every transaction. If a transaction has a start record and no commit record and no abort record, then by atomicity it either works entirely or not at all — it has not committed, so all of its operations must be undone. If a transaction has a commit record, its operations must be redone — re-established on the database.

17.3.9 Q&A: Isolation Levels and the Question-Setter's Psychology

A student reports seeing questions about isolation levels in old question papers, a topic that seems to be covered nowhere in the syllabus material or in the sessions, and asks whether it will appear in the examination.

The answer then turns into a broader lesson about preparation. You do not need a PhD in psychology to read the question-setter's psychology. The best lawyers prepare by understanding the judge: how they refer to things, what they focus on, which of their top logics this case will trigger — not a plain template, but a presentation tailored to the person and the situation. The same applies to an examination: the question is set by a person, the topics that person focuses upon get the questions. Understand how the course material was presented — what was covered and how deeply — and you can predict both the questions and the expected depth. A general "plain vanilla" preparation covers everything; a smart preparation weights the areas the setter visibly emphasized. This transfers directly to professional life: your immediate supervisor or manager is the one you report to, and reports shaped for a previous manager may not fit the current one. When a manager asks "have you covered this portion?", the answer is already in your memory of what was covered and how — and if other instructors taught something in more detail in earlier years, their exams expected that detail; here the depth expectation matches what was actually taught.

17.4 Checkpointing: Where Does Recovery Begin?

17.4.1 The Problem: How Far Back Do We Scan?

The basic recovery algorithm has a fatal practical flaw, and the session builds it up with an extreme example. The system started working years ago — say the application went live in 2001, and the failure happens in May 2024. Does recovery scan the log all the way back to 2001, redoing and undoing decades of operations? Consider an application like IRCTC — train ticket booking — where people arrive rampantly day and night. Every second there are hundreds of transactions, every hour hundreds at least; in 10 or 20 days there are thousands; after a year, an order of magnitude higher. Two years in, a failure happens — a small power failure, or a big hardware failure, since hardware always fails eventually. Where does the scan start? Ten years back? Three years back? One second back?

Ask ten people and you get ten answers: one second, one hour, one day, five days, fourteen days, one year. Nobody is sure when the last point was that everything in memory got written back to secondary storage — whether a given committed portion has reached the disk or not. The recovery algorithm is a computer program, and a computer program needs a deterministic boundary: "after this point, I will not look at the log files". The session's analogy is an exam deadline: when a quiz has a hard closing time — 10:59 AM — there must be a clear rule about when responses stop being accepted, one that is followed every single time, the way legal precedent is followed by every lower court once a higher court rules. What is that boundary for recovery? The answer is the checkpoint.

17.4.2 The Checkpoint Boundary

A checkpoint is a point in time at which everything that was happening up to then has been written back to secondary storage. If a checkpoint exists, anything before it is already safely on disk and needs no attention. The recovery scan then starts at the checkpoint — not at the beginning of the log, and not at an arbitrary time. Transactions that finished before the checkpoint are ignored entirely: their work is already persisted. Only transactions that were still running at the checkpoint, or that started after it, need to be examined. And the scan is still safe even when a redo is unnecessary — because redo operations are idempotent, redoing a change whose value already reached the disk is harmless. The checkpoint converts the unbounded "where do I start?" question into a fixed, known boundary, and it is deterministic: every failure, the same rule applies.

17.4.3 Worked Example: One Checkpoint, Four Transactions

The first worked case reads a log that contains a checkpoint. Suppose transaction $T_1$ was running and finished before the checkpoint. Then $T_1$ needs no attention — everything $T_1$ did was written back before the checkpoint, so its log records are ignored, and so are the records of everything before it. That is the saving the checkpoint produces.

At the checkpoint itself, $T_2$ had started, with some operations in progress. $T_2$ committed after the checkpoint. Its operations must be redone — even though the values may already be on disk, redoing them is safe because the operations are idempotent, so the redo is mandatory but harmless. Transaction $T_3$ also committed after the checkpoint; the same logic applies: redo $T_3$'s operations. Transaction $T_4$ started but never committed — the log has a start and no commit — so undo $T_4$'s operations. In one line: redo the committed $T_2$ and $T_3$, undo the uncommitted $T_4$, and ignore the pre-checkpoint $T_1$ entirely.

   T1 (finished)       checkpoint        failure
      |                    |                 |
      |  T2 (in progress)  |--------- commit(T2)
      |                    |  T3 ---- commit(T3)
      |                    |  T4 ---------- (no commit)
      v                    v                 v
   <----------------  scanned area  ----------------->

17.4.4 Worked Example: The Undo and Redo Buckets

The second worked case is the "beautiful case" that shows the standard two-bucket procedure. The log contains a checkpoint that records two transactions in operation at that moment: $T_0$ and $T_1$. The procedure starts by placing both in the undo bucket: by default, every transaction active at the checkpoint is assumed to need undoing. As the scan moves forward through the log, each transaction's fate is decided by its commit record.

• $T_1$'s commit appears in the log → move $T_1$ out of the undo bucket and into the redo bucket. All of $T_1$'s operations are redone.
• $T_0$ has a start and no commit → stays in the undo bucket. Its operations are undone, and the log is updated with a $T_0$ abort record.
• $T_2$ starts after the checkpoint and never commits → it was never in the original bucket list, but the same rule applies: it joins the undo bucket. Its operations are undone, and the log gets a $T_2$ abort record.

The final state: $\text{redo bucket}=\{T_1\}$, $\text{undo bucket}=\{T_0,T_2\}$. The recovery does exactly that — redoes $T_1$'s operations, undoes $T_0$'s and $T_2$'s operations, and writes the abort records for $T_0$ and $T_2$ so the log itself reflects the recovery.

The reaction to the case is a teaching moment in itself: "this is so simple". The claim is that an entire database course is full of topics that seem mysterious until explained once — the way 2NF sounds impossible until someone walks you through it — and checkpointing is exactly that kind of topic: once the boundary is clear, the bucket procedure is trivial.

17.4.5 Q&A: Restarting After an Abort

Two questions close the checkpoint discussion, and both are about what happens after a failure is handled.

17.5 SQL vs NoSQL, Sharding, and the Modern Database

17.5.1 The Pre-2000s World: One Database, Manageable Data

The final topic of the session is framed as a bonus — explicitly stated to be outside the examination — but insisted to matter for real life. Before the 2000s, the database world was simple: a single database in a single location, queried in the network format, and the whole arrangement was manageable. Data was not bombarding in volume; the workload was modest. Then the world changed. Hotmail and Gmail arrived, Yahoo mail was already there, and mail started coming in big proportions — the SMTP protocol moved messages, but the database behind the mail service itself became very high-volume. Shopping applications followed, and so did social media applications in the style of Orkut and, soon after, YouTube. Each of these pushed more data at the database than the old single-store arrangement could comfortably absorb.

17.5.2 Volume, Variety, Velocity: The Three Limits

The old relational database had constraints that the new world exposed. The data it was designed for was limited: mostly text that could be stored, without variety — no audio, no video, no multimedia combinations — and it was not increasing in volume. The new world pushes in three directions at once: volume (data increasing), variety (many kinds of data), and velocity (the pace at which data arrives).

In that situation, following ACID strictly is difficult, and storing everything in a single relational database in one location is not easy either. Even if a miraculously scalable format existed for storage, the application layer creates a second problem: relational queries depend on joins, and joins need every table. A relational application cannot work unless all the tables and relations are available together.

17.5.3 The Replication Argument and MongoDB's Rise

The session walks through a concrete replication scenario using the social media application built in the course. The database sits locally, with replication. But the decision taken was to replicate the entire database at a remote site: if a query arrives, the remote site cannot answer it unless all the tables and all the relations are present there, because the joins need everything. That is the honest cost of relational replication — you replicate the whole database or nothing works. The audience is asked to agree: unless everything is replicated, queries will not work. That is precisely the situation that explains why MongoDB and the NoSQL family came into existence. The new approach lets go of the rigid table boundaries: all the data about movies, students, universities — and data that does not even exist yet in the old schema — sits in a single flexible format. A document holds the name, roll number, ID, phone, marks, and courses together in one file, and replication becomes easy because you do not need to replicate every relational constraint — just the documents. The application becomes much simpler to run and much easier to scale.

Sense-check: the relational replica needed every table because joins are global operations; the document replica needs only the documents because every document is self-contained. The join wall is what made relational replication heavy; self-containment is what makes NoSQL replication light.

17.5.4 NoSQL Formats and Eventual Consistency

NoSQL is not one format — it is any format that is not the relational one: MongoDB as a document format, column-based stores, graph-based stores, and document stores like Cassandra. Each format has its own advantages, and the primary reason the family exists in real life is that the relational approach took a lot of time to establish and to maintain, and its strict ACID guarantees are not always needed. Sometimes eventual consistency is fine: the application accepts that in between, data may be momentarily inconsistent, as long as it becomes consistent eventually. That trade — weaker guarantees for easier scale — is the deal that motivated the migration to the new stores.

Real-world: the eventual-consistency trade is the same one made by modern social feeds, notification counters, and recommendation caches: a few moments of staleness are acceptable; losing the ability to scale is not. Your feed can be five seconds behind a friend's post — you will not notice — but the service cannot afford to make every feed read wait for every replica on the planet.

17.5.5 Sharding: Facebook's Path to Scale

The big-name technique in this story is sharding, which Facebook made famous. The idea: split the entire database into portions, and keep each portion replicated — in the picture drawn here, three replica copies, with a master and a sub-replica arrangement where the directory knows where the actual copy of each portion lives. A query from Singapore, one from Switzerland, one from India: each hits the database, is told where the actual data is, goes to that portion, and is served locally. The portion knows where its data is, and everything related to serving the query resides in that portion.

Sharding delivered scalability "from decent to great", and the mechanism that made it possible is the same letting-go of rigid tables: data is sorted and dumped across portions in JSON format. Because JSON is a plain, replicable representation, the portions can be replicated freely, and because replication works, the application becomes scalable. That is the modern pattern: new-age databases that people work with day in and day out are built on this trade.

17.5.6 PL/SQL: Object Style Over Procedural Style

Alongside the new stores, the relational world itself evolved. PL/SQL and object-based formats allow a more object-style way of working with databases — the analogy given is the migration from C to Java: procedural style to object-oriented style, more natural for modern programmers. The claim is reassuring: the heart remains the same. PL/SQL is still SQL, the relational core still sits underneath, and the object style is a wrapper around the same ideas.

17.5.7 Q&A: Is Sharding in the Exam?

The status of the whole NoSQL discussion is settled with a direct question from a student.

Exam note: sharding, NoSQL formats, and the SQL-versus-NoSQL comparison are not part of the comprehensive examination. The topic list that is examinable was given in section 17.1.

Exam Guidance Summary

The comprehensive examination guidance, gathered in one place. Everything here was stated in the session; nothing has been added.

• Scope: whatever is in the handout is part of the comprehensive examination — including topics from live sessions and recorded sessions. Nothing in the handout is exempt.
• Examinable topics: normalization, ER model, SQL basics, relational algebra, relational calculus, B-tree and B+ tree indexing (must be practiced), database security and access control, and recovery. Lab-session query work is also part of it.
• Not in the exam: query optimization (it exists in the on-campus offering only), and the NoSQL material from section 17.5 — sharding, MongoDB, and the SQL-versus-NoSQL comparison.
• Where the questions will lean: topics after the mid-semester were assessed only feebly in the quizzes, so expect more opportunity to score there. Self-study topics are at worst about 10 marks out of 40 — covered topics carry the rest.
• Open book: the examination is open book, so know where every topic lives in a textbook and in the references. Use bookmarks and ear-tags; practice a previous-year question paper.
• Grading: relative grading, with a 50 percent average guideline for the course. Individual marks only matter relative to the class.
• Uncertain topics (e.g., isolation levels): check the handout first, then the live and recorded sessions; bookmark the topic in the textbook; in the exam hall, use the book's index and study on the spot before attempting.
• Answer-writing strategy: for every question, write one line of what you understand the question to mean; then the background theory in one or two lines; then your application of it; then consequences and your verdict. Write all assumptions out in full. Articulation is what earns partial marks.
• SQL questions: evaluated as essentially correct or incorrect — a semicolon or a comma can change whether a query runs. Multiple correct solutions exist (nested queries, joins, join styles); what matters is that the query actually works and that your thought process is visible.
• Planning: seven days, one hour per day, with at least one session on a previous-year question paper; then bookmark-and-navigate.

Key Industry Applications

The real-world names behind this lecture's technical content. Each entry ties one concept to where it actually shows up in industry.

• IRCTC-style ticket booking: the running example of extreme transaction load — hundreds of transactions per second — used to motivate both concurrency control and checkpointing.
• Social media applications: the course's own small social app is the example database for the log-file worked cases and the replication argument.
• Hotmail, Gmail, Yahoo mail: the post-2000 explosion in mail data that outgrew the single-store relational arrangement; SMTP moves the mail, the database behind it is huge.
• Shopping applications and Orkut, YouTube: the platforms that pushed volume, variety, and velocity past the old limits.
• MongoDB and the NoSQL family: document, column-based, and graph formats (Cassandra included) that trade strict ACID for scale and ease of replication.
• Facebook sharding: splitting the database into replicated portions with location-aware routing — queries served from the nearest copy, in JSON, enabling scalability.
• PL/SQL: object-style database programming, compared to the C-to-Java migration, with the relational heart unchanged.
• Journal-style logging: the log-file format $⟨T_i,X,v_{\text{old}},v_{\text{new}}⟩$ is the same audit-trail idea used in financial accounting and file-system journaling.
• Stable storage by replication: the vault analogy — multiple copies at different places surviving power loss, hardware failure, and natural disaster.
• Undo/redo in everyday software: Ctrl-Z and Ctrl-Y in word processors as the everyday face of the undo-redo recovery operations.
• Word-of-mouth professional practice: the problem-consequence-options-evaluation-recommendation framing taught for approaching any authority — exam teams, managers, or clients.