Data Management for Machine Learning

Privacy Governance and Responsible Machine Learning

Published: 2026-09-15

Level: postgraduate

Audience: Postgraduate students studying data management for machine learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

LLM data pipelines, vector stores, and retrieval with context assembly — covered in Lecture 14 (Data Privacy, Governance and LLM Data Pipelines)
Data privacy foundations: breach, PII, and the six compliance pillars — covered in Lecture 14 (Data Privacy, Governance and LLM Data Pipelines)
GDPR, India’s Digital Personal Data Protection Act, and HIPAA duties — covered in Lecture 14 (Data Privacy, Governance and LLM Data Pipelines)
Privacy-preserving machine learning techniques — covered in Lecture 14 (Data Privacy, Governance and LLM Data Pipelines)
Feature engineering across ML, deep learning, and LLM projects — covered in Lecture 4 (Data Pipelines, Big Data Systems, and Feature Engineering)
Distributed processing with MapReduce and HDFS, plus Spark — covered in Lectures 4, 12 (Big Data Systems: Distributed Storage and Distributed Processing), and 13 (Big Data Ecosystems, Cloud Platforms, and LLM Pipelines)
Metadata systems and k-fold experimentation — covered in Lecture 11 (Machine Learning Experimentation and Metadata)
Streaming ingestion with Kafka and lake, warehouse, and mart storage layers — covered in Lectures 4 and 13

Privacy Governance and Responsible Machine Learning

This session links two worlds that students often see as separate: building fast machine learning pipelines, and protecting the people whose data flows through them. It opens with two student builds — a payment failure triage pipeline and a medical grounding pipeline — then uses the temperature control to show how a single prompt setting changes output discipline. The middle blocks set the legal floor: breach and PII, the six compliance pillars, GDPR with ten duties, the Indian Digital Personal Data Protection Act, and HIPAA with SPI and PHI. The closing blocks turn law into engineering: four privacy preserving learning methods, anonymization with k-anonymity and synthetic data, a full generative AI stack with security at every layer, and a map for scoring on the exam.

15.1 Student Project Showcase: Payment Failures and Medical Knowledge Graphs

15.1.1 Root Cause Analysis of UPI Payment Failures With LangChain and LLM

A payment app fails at dinner time. Millions of people tap pay at once. Which failures are a dead phone network, which are a down bank server, and which are fraud blocks? Answering that by hand at national scale is not possible.

Root cause analysis, shortened to RCA, means tracing a visible failure back to the first cause that can be fixed. An end-to-end pipeline means one connected chain from raw logs to a dashboard a manager can read. A large language model, written LLM, is the reader in the middle that sorts messy log text into neat classes.

The everyday picture is a hospital emergency desk. Patients arrive with one line each: fever, cut, cough. A triage nurse sorts them in seconds into wards. Here the patients are failed payments, the nurse is a LangChain chain with an LLM, and the wards are 15 failure classes. The picture breaks at one point: a nurse uses judgment from years of care, while the chain follows the prompt and the schema it was given.

UPI scale and the failure math. UPI is the national instant payment rail discussed in class. Stated scale is over 131 billion transactions worth about 199 lakh crore, with a failure rate of about $3%$ . Here $3%$ means 3 failed payments out of every 100.

The yearly failed count $F$ follows:

$F = N \times r$

where $N$ is total transactions and $r$ is the failure share. With $N = 131 \times 1 0^{9}$ and $r = 0.03$ :

$F = 131 \times 1 0^{9} \times 0.03 = 3.93 \times 1 0^{9}$

So about 3.93 billion failures per year. Divided by 365 days:

$F_{d a y} = \frac{3.93 \times 1 0 ^{9}}{365} \approx 10.8 \times 1 0^{6}$

About 10.8 million failed payments per day. That number is why manual triage cannot keep up and why zero-shot categorization plus a dashboard matters.

The pipeline has five stages in order. First, a LangChain chain performs zero-shot failure categorization into $15$ distinct failure classes, where $15$ is the count of classes and zero-shot means no per-class training rows were supplied. Second, a structured 5 Whys analysis asks why at each level, and a fishbone (Ishikawa) diagram groups causes into bones such as network, bank, app, user, and fraud. The 5 Whys is the standard five-level why chain used in quality work. Third, pandas DataFrames carry the records through analysis, where pandas is the Python table library and a DataFrame is its row-and-column table. Fourth, a Streamlit analytics dashboard presents 8 interactive visualizations, where Streamlit is the Python dashboard tool and 8 is the count of views. Fifth, a machine learning baseline sits after the LLM stage so scores can be set against classical models.

A sketch of the visual helps. The horizontal axis is pipeline time from log entry on the left to dashboard on the right. Boxes rise in the middle for the LLM categorization and the 5 Whys plus fishbone step. The takeaway in one line: messy text enters on the left, fixed classes and counted causes leave on the right.

Scope: This RCA pattern fits high-volume event streams with short log lines and a fixed class list. Assumption: log fields hold enough signal to sort, and the 15 classes cover the real failures. When logs are empty or a new failure mode appears outside the 15, the chain mis-sorts and the counts mislead until the class list and prompt are revised.

Pitfalls. Treating the LLM label as ground truth without sampling and checking against logs. Letting the class list drift so two classes overlap and the same failure lands in both. Skipping the baseline, so nobody knows whether the LLM beats a simple classifier enough to pay its cost.

Worked triage count. Take one day with $N_{d a y} = 360 \times 1 0^{6}$ attempts and $r = 0.03$ .

$F_{d a y} = 360 \times 1 0^{6} \times 0.03 = 10.8 \times 1 0^{6}$

If 22% of those are network timeout, the timeout load is:

$10.8 \times 1 0^{6} \times 0.22 = 2.376 \times 1 0^{6}$

2.376 million timeouts in one day. Sense-check: a single team reading 200 cases per day would need about 11,880 people-days to clear only the timeouts, so automation is required.

Recap: UPI scale turns a small $3%$ rate into about 10.8 million daily failures, sorted by a LangChain plus LLM chain into 15 classes with 5 Whys and Ishikawa review, pandas flow, and an 8-view Streamlit dashboard. Bridge: the next step is what those 15 classes are and how features and severity rules are built from them.

UPI interfaces process national-scale payment volume, and the same RCA pattern fits any high-volume transaction system that needs fast failure triage, from card rails to ticket booking.

15.1.2 Failure Categories, Feature Engineering, and Severity Rules

Feature engineering means building the input columns a model actually reads from raw logs. Severity means the ground rule that marks which failure classes need instant action and which can wait.

Concrete failure categories named in class are insufficient funds, network timeout, bank server outage, wrong MPIN, fraud detection, transaction limit breach, UPI app timeout, account block, and duplicate transaction. Given transaction logs, the pipeline assigns each failure to one of these categories and then runs feature engineering over the log fields: hour of day, bank code, app version, error code, retry count, amount bucket, and device channel.

Runnable chain shape. The orchestration uses a runnable chain built from a passthrough step plus a prompt plus a string output parser. In words, the chain takes the record, formats it through the prompt, runs the model, parses the string output, and extracts the category. Once the category is extracted, a program analyzes it with a fixed LLM temperature setting, writes a JSON file, pulls the records with pandas, answers questions over them, and serves the output on the Streamlit dashboard. Here JSON is the plain text record format, pandas is the table step, and Streamlit is the display step.

Feature engineering gets strong emphasis: whether the project is machine learning, deep learning, or LLM based, the features must be built with care. The project also defines ground rules for severity, marking which categories count as severe or high severity, for example bank server outage and fraud detection rank above wrong MPIN retries.

Worked severity pass. Take 1,000 failed records. Suppose the chain labels 300 as network timeout, 180 as insufficient funds, 120 as bank server outage, 90 as wrong MPIN, 60 as fraud detection, and 250 across the rest. If severe means bank outage plus fraud detection, the severe share $S$ is:

$S = \frac{120 + 60}{1000} = \frac{180}{1000} = 0.18$

18% severe. Sense-check: fewer than one in five need paging, the rest queue for batch review, which matches an operations desk that pages only on outage or fraud.

Scope: severity rules hold only for the purpose they were set for. Assumption: the label set is stable and the log schema keeps the same fields. When a bank adds a new error code or an app version changes field names, features must be rebuilt or counts shift for no real reason.

Exam note: expect questions that ask what feature engineering means inside an LLM project, not just inside classical ML. The answer: selecting log fields, bucketing amounts and times, encoding bank and error codes, and passing only the fields the prompt and baseline need.

15.1.3 Model Comparison, Prompt Design, and Explainability

Role prompting means the prompt first states the role the model must play, such as payments triage analyst. JSON schema compliance means every model reply matches the fixed JSON shape with no extra keys and no broken brackets. Explainable AI means methods that show which inputs moved a decision.

Model choice is not only price per token. A cheap model that breaks the JSON schema on every tenth call costs more in repair code, retries, and wrong dashboard counts than a pricier model that returns clean JSON each time.

The project compares Claude Sonnet, GPT-4o, GPT-4o mini, and LLaMA, each with its provider and its cost per million input tokens. Claude Sonnet was selected as the default because of stronger JSON schema compliance even though cost mattered. The lesson holds: weigh price against output discipline.

Prompt design follows named principles. Role prompting comes first. Named prompt styles kept as labels in class are GO STAR style prompts, CARE style prompts, and a FACT style prompt, followed by a strict JSON schema and a fixed temperature. The exact expansions of those style names were not spelled out, so they are used here as labels for the prompt patterns the team tried.

The ML baseline uses logistic regression and random forest, with random forest and XGBoost named for the modeling stage. Here logistic regression is the linear classifier for yes-or-no style odds, random forest is a vote across many decision trees, and XGBoost is boosted trees trained in sequence. A final recommendation is to add SHAP based explainable AI, where SHAP values split a prediction among input features so the team can check which features drove each decision and whether the right features were selected.

A side-by-side contrast helps. Claude Sonnet leads on schema discipline and suits the dashboard path. GPT-4o is strong all round and suits mixed text work. GPT-4o mini costs less and suits high-volume first passes with a check step after. LLaMA suits local or tuned builds where data must stay inside. Pick the model that meets the schema bar first, then pick by price.

Worked cost versus repair trade. Suppose one million records need labels. Model L costs 0.50 per million tokens at 500 tokens per record, so input cost is $0.50 \times 500 = 250$ . Model H costs 3.00 per million, so input cost is $3.00 \times 500 = 1500$ . If Model L breaks schema on 12% of calls and each break costs 0.02 in retry plus fix time, repair is $1, 000, 000 \times 0.12 \times 0.02 = 2400$ . Total for L is $250 + 2400 = 2650$ , above H at 1500 with near-zero breaks. Cheaper per token can cost more end to end. Sense-check: schema breaks scale with volume, so discipline matters more as volume grows.

Pitfalls. Tuning temperature for creativity on a classification step, which breaks the schema. Judging models only on a demo of 20 rows instead of a full error sample with schema checks. Adding SHAP at the end with leaky features that the model should never have seen.

Recap: compare on price plus JSON discipline, prompt with role plus strict schema plus fixed temperature, baseline with logistic regression, random forest, and XGBoost, and check with SHAP. Bridge: the same discipline carries to the medical build, where a wrong label harms far more than a dashboard count.

15.1.4 Medical Grounding With SNOMED Concepts and a Second Project

Grounding means tying a model answer to a trusted source so it cannot invent facts freely. SNOMED CT is the clinical terminology that holds medical concept identifiers. A property graph, which is a knowledge graph, stores those concepts as linked nodes with named relations.

A second showcase uses a fused LLM over medical data. It draws on the SNOMED CT terminology and reasons over a property graph to ground predictions and check truth. The stack named alongside includes in-memory networks, graph LLMs, coding assistants, and an orchestrator, plus a literature survey the builders had to study. Here the orchestrator fans work across tools, the graph LLM reads linked concepts, and the survey records what prior work tried.

Picture a map wall with pins and strings. Each pin is a SNOMED CT concept such as a symptom or drug. Each string is a relation such as treats or causes. The LLM may draft a sentence, but each medical claim must touch a pin and follow a string. The takeaway in one line: no pin and no string means no claim.

Why grounding fights hallucination. An LLM alone predicts likely words. A curated graph holds checked links. Retrieval first pulls the linked subgraph for the question, then generation must stay inside it. When the model tries to invent a drug link that has no edge in the graph, the check step rejects it before display.

Worked grounding check. A draft says drug D treats symptom S. Retrieval pulls all treat edges from D. If the edge set is {D treats S1, D treats S2} and S is absent, the claim fails the check and is blocked or sent for human review. Blocked answer beats a fluent wrong answer. Sense-check: this matches the safety bar in care settings, where a wrong name or dose harms patients.

Scope: grounding is only as good as the terminology version and graph coverage. Assumption: SNOMED CT IDs are current and the graph links the needed relations. Rare terms or new drugs outside the graph need an update path, not a forced guess.

Recap: curated SNOMED CT identifiers plus a property knowledge graph hold a fused LLM to checked truth. Bridge: with both showcases set, the next block isolates the one dial that most changed output discipline: temperature.

Grounding with a curated knowledge graph fights hallucination in care settings where a wrong answer harms patients, and the same pattern fits finance and law where claims must cite a source.

15.1.5 Student Questions and Answers

Q: A student is asked to read the project aloud for the class, and there is confusion about who should share their screen.

A: The class works through it live: one student reads the UPI root cause material while screen sharing is sorted out, and the discussion moves forward through the architecture, the UPI payment failure pipeline with zero-shot categorization into 15 categories, the model comparison table where Claude Sonnet leads on JSON schema compliance, and the code together. The flow keeps the opening project showcase and the model table in source order: pipeline first, table second.

15.2 LLM Selection, Prompting, and the Temperature Control

15.2.1 Why Temperature Matters in Prompting

Ask the same factual question twice. At low temperature you get the same right answer twice. At high temperature you get two lively but different answers, one of which may be wrong. Which behavior do you want for code and facts?

Temperature, written $T$ , is the prompt engineering control that decides how deterministic the model is. A deterministic reply means the same input gives the same output each run. The most statistically likely completion means the top-probability next tokens win each step.

Think of a vending machine versus a story dice cup. Low temperature is the vending machine: same button, same drink. High temperature is the dice cup: same shake, new tale. The mapping breaks at one point: a vending machine never surprises, while even $T = 0$ can still err if the prompt or data is wrong.

Three bands. Low covers values such as $T = 0$ and $T = 0.2$ , where $T$ is the temperature parameter, a non-negative scalar, with $T = 0$ the fully deterministic end. Medium is the warm middle. High is the novel and creative end. The verbal anchors stay fixed: most statistically likely completion belongs with the low band, warm with the middle band, and novel, creative, diverse, unexpected with the high band.

Picking the wrong temperature is a common silent bug: creative settings on factual work invent facts, while fully deterministic settings on creative work return flat and repeated ideas. Temperature determinism for the jet engine task and for code tasks is the worked anchor for the low band.

Picture the band as a slider from left to right. Left end is ice: fixed, exact, narrow. Middle is room warmth: shaped but flexible. Right end is heat shimmer: loose, varied, surprising. Horizontal axis is $T$ from 0 upward. The takeaway in one line: move left for one right answer, right for many fresh answers.

Scope: temperature shapes sampling only. Assumption: the model knows the fact and the prompt states the task. When the model lacks the fact, low temperature returns the same wrong answer each time with high confidence, so retrieval or a better prompt must fix the gap first.

Pitfalls. Leaving temperature high for extraction, classification, and JSON output, then blaming the schema. Pinning temperature at zero for brainstorming and concluding the model has no ideas. Changing temperature between runs while testing prompts, so gains cannot be traced to prompt or dial.

Worked band check. Prompt: define UPI failure category for wrong MPIN. At $T = 0$ three runs return wrong MPIN three times: stable and sortable. At a high setting three runs return wrong MPIN, wrong PIN retry story, and a new label invented on the spot: varied but one breaks the 15-class schema. Low wins when the schema must hold. Sense-check: matches the production rule to pin near zero for extraction and classification.

Recap: $T = 0$ and $T = 0.2$ give the most likely completion for exact work; warm middle gives flexible drafts; high gives novel and diverse ideas. Bridge: the next block maps each band to concrete jobs with the jet engine setting as proof.

15.2.2 Worked Guide: Which Band for Which Job

Low, predictable, focused, deterministic work belongs at the bottom of the range. Use low temperature for Java code, C++ code, basic mathematics, factual question answering, and technical documentation. The rule of thumb is simple: when exactly one right answer exists, turn the temperature down.

Medium fits mixed, semi-structured material such as emails and blog posts, where some structure exists but wording can vary. High fits novels and creative writing, where diversity is the goal rather than a defect.

Jet engine setting. To write a technical explanation of how a jet engine works with high factual accuracy, set the temperature to $T = 0.1$ . Here $T = 0.1$ is a low value near the deterministic end, chosen so accuracy stays high. Students were asked to screenshot the temperature guide and share it in the class group because it is used again and again across tasks.

A compact table keeps the mapping ready for the exam:

Band	Values	Use for	Example
Low	$T = 0$ , $T = 0.1$ , $T = 0.2$	Code, math, facts, docs, extraction	Java code, jet engine explainer at $T = 0.1$
Medium	warm middle	Emails, blogs	Draft update with own voice
High	creative end	Novels, ideas	New story opening, varied slogans

When to pick which in one line: exact output means low, shaped draft means medium, fresh variety means high.

Worked task sort. Sort five jobs: Java function, C++ fix, factual Q and A, product email, short novel scene. Java, C++ fix, and factual Q and A go low at $T = 0$ to $T = 0.2$ . The jet engine explainer joins them at $T = 0.1$ . Email goes medium. Novel scene goes high. Three low, one medium, one high. Sense-check: the three low jobs each have one right answer, the email has a shape with free wording, the scene wants surprise.

Scope: bands are guides, not hard walls. Assumption: sampling method stays fixed while $T$ moves. Very long outputs drift even at low $T$ , so split long factual docs into short pinned sections with checks.

Production teams pin temperature near zero for extraction, classification, and code tasks, and raise it only for brainstorming and draft variety.

Exam note: temperature bands are directly testable. Low $T = 0$ , $T = 0.1$ , $T = 0.2$ for deterministic factual work including the jet engine explanation, medium warm for semi-structured writing such as emails and blogs, high novel and creative for stories. Bridge: with output control set, the session turns to what must be protected in the first place: breach, PII, and pillars.

15.2.3 Student Questions and Answers

Q: What is the use of temperature in prompt engineering and in LLM calls?

A: Temperature controls determinism. Low values such as $T = 0$ or $T = 0.2$ make the model return the most likely completion, which suits code, mathematics, facts, and documentation including Java code, C++ code, basic mathematics, factual question answering, and technical documentation. Medium values suit semi-structured writing such as emails and blogs. High values suit creative and novel output such as novels and creative writing. For the jet engine explanation task, the guidance is $T = 0.1$ so accuracy stays high with a factual jet engine answer.

15.3 Data Privacy Foundations: Breach, PII, and Compliance Pillars

15.3.1 Breach, Information Privacy, and Why Businesses Collect Data

One leaked file can undo years of trust. Why do businesses keep collecting so much, and what duty comes with each stored field?

Data breach means protected items leave trusted hands without permission. Data privacy, also called information privacy, means keeping each person's information in their own control. Compliance means following the laws and standards that govern collection and storage.

The chain runs breach to privacy to compliance: because breaches happen, privacy rules exist, and because privacy rules exist, every data project must obey them. Large volumes of data bring a large risk of data breach, so volume itself raises the duty of care.

Businesses collect data from users on a regular basis and store it for service, safety, billing, and growth. Each purpose must be stated, each store must be guarded, and each move must be logged. A team that cannot say why it holds a field should not hold it.

Collect, store, answer. Collection is the intake act. Storage is the holding act with access rules. Compliance is the proof act: show purpose, consent, limits, and guards for each item. When any link is missing, breach risk and legal risk rise together.

Picture a warehouse ledger. Left column lists what arrived, middle column lists where it sits and who holds a key, right column lists the signed permission slip. The takeaway in one line: no slip means no stock.

Scope: this foundation covers planning and duty, not the math of hiding. Assumption: the team can list its sources and stores. When shadow copies live in laptops and chat threads, the ledger is fiction until those copies are found and ruled in or out.

Worked ledger row. Field: phone number for delivery updates. Purpose: contact on delay. Store: orders table with role access. Permission: checkout consent tick. Retention: order window plus 30 days. One row ties purpose to store to permission to time. Sense-check: any field without such a row is a candidate for removal.

Recap: breach prompts privacy, privacy demands compliance, and every held field needs a stated purpose and guard. Bridge: the next step names the fields that trigger all of this: PII.

15.3.2 Personal Identifiable Information

Personal Identifiable Information, shortened to PII, means any item that picks one person out of a crowd, such as a name, an identity number, or contact details.

PII as the start line. The class states the expansion together: when asked what PII means, the answer given is Personal Identifiable Information. This definition matters for the exam because later sections contrast PII with sensitive variants. Every later protection technique in this session starts by finding the PII first: locate it, mark it, then decide consent, minimization, security, and rights around it.

Think of a crowd photo with one red circle. PII is whatever draws that circle: face, badge number, phone on screen. Remove the circle makers and the photo shows a crowd but no person. The picture breaks if two weak clues join: neither names anyone alone, but together they single out one face, so marking must catch combinations too.

Worked PII hunt. Table columns: name, age band, city, order total. Name is direct PII. Age band plus city plus order total can turn quasi-identifying in a small town sample. Mark name first, then test the combo. Sense-check: matches the rule that de-identification must test joins, not single columns alone.

Scope: PII labels shift with context. Assumption: the team knows the join keys in its own lake plus likely outside sets. A field that looks safe alone can identify when linked, so review must repeat as new sources arrive.

Recap: PII is the finder step; all guards hang off it. Bridge: six pillars then say what to do once PII is found.

15.3.3 The Six Pillars That Shape Every Privacy Question

A single diagram carries six ideas that answer most exam questions on perspective: accountability, subject rights, consent, transparency, data minimization, and security.

Six in plain words. Consent sits inside data protection as the right to share or not share. Transparency means saying, for each specific item, what was collected, how it is used, and how it is shared, in a two-way exchange that lets the person make an informed choice, for example whether an item serves marketing or serves the core service. Data minimization means sharing only what the purpose needs and nothing more. Security covers encryption and access control. Accountability means the people who handle data answer for their actions under data protection law. Subject rights decide who may correct or delete an item: a doctor, a patient, or the hospital as an entity hold different rights, and some items can be edited or deleted while others cannot.

Think of borrowing a bike. Consent is asking first. Transparency is saying where you will ride and for how long. Minimization is taking only the bike, not the garage keys. Security is locking it outside the shop. Accountability is owning damage. Subject rights say who may lend, recall, or scrap the bike when roles differ.

A shared drive full of course folders shows why access control matters. When sharing one item such as a transformer code file with one classmate, grant only that file, only to that person, only with read access, and nothing wider. That grant shows minimization plus security plus accountability in one click: least item, least person, least power, with a logged owner.

Worked share grant. Share transformer code file with one classmate. Steps: select file only, add named person only, set read only, set expiry, log grantor and time. One file, one person, read only. Sense-check: any wider grant breaks minimization even if nothing leaks yet.

Scope: pillars guide judgment across regimes. Assumption: roles and rights are written down. Doctor, patient, and hospital hold different edit and delete rights, so a delete request must check whose right applies to that item before acting.

Exam note: the six pillars read accountability, subject rights, consent, transparency, data minimization, and security. Pair each with the shared drive grant and the doctor-patient-hospital rights split for full marks on perspective questions.

15.3.4 Student Questions and Answers

Q: What is PII, and what is the meaning of the term?

A: PII is Personal Identifiable Information. It is the starting point for every protection decision: find the fields that identify a person such as name, identity number, and contact details, then decide consent, minimization, security, and rights around them. The shared drive story shows the same duty in action: grant a single transformer code file to one classmate with read access only, which keeps identifiable and licensed material fenced while work moves ahead.

15.5 Sensitive Information: SPI, PHI, and HIPAA Rules

15.5.1 PII Versus SPI Versus PHI

Three similar letters cause most mix-ups on this paper. Which label fits a phone number, which fits a live GPS pin, and which fits a smoke habit noted by a doctor?

PII is Personal Identifiable Information. SPI is sensitive personal information, a special category introduced with the California Privacy Act. PHI is patient health information, also stated as protected health information, guarded under health law.

Three labels, three homes. PII is general identity that singles out a person. SPI is the sensitive slice such as precise location, racial origin, and health facts that needs higher care. PHI is health-side items such as social history like whether a person drinks or smokes, which sits with the physician first, not with the world. Exam note: a likely question asks which act deals with which label, so tie each label to its regime: general identity to PII practice, sensitive traits to the California category, and health items to HIPAA.

Think of three envelopes. PII is a plain envelope with a name tag. SPI is a sealed envelope with a red stripe. PHI is a clinic file that only care staff may open, with social history inside. Same paper, very different handling.

Worked label sort. Name plus phone is PII. Live GPS pin plus racial origin note is SPI. Drink-or-smoke note in a clinic chart is PHI and social history under PHI. Sort first, then guard by envelope. Sense-check: the smoking note gets clinic-grade care even though smoking alone sounds ordinary outside care.

Scope: labels can stack. Assumption: one item can be both PII and SPI or PHI at once. A named health note needs both identity guards and health-law guards, not one or the other.

Recap: PII names, SPI stings, PHI heals under lock. Bridge: next lists what counts as sensitive with the location case as proof.

15.5.2 What Counts as Sensitive

Sensitive items include precise geolocation, racial origin, and health facts.

Knowing one classmate's exact longitude, latitude, and GPS coordinates is sensitive because it can reveal where that person physically is at any moment. The class jokes that tracking a classmate to a theatre through a leaked location would be a misuse, then makes the serious point: leaked location can threaten safety, and anyone harmed by misuse after consent expired could seek remedy. That is why sharing a live location through a chat app offers a short window such as half an hour or one hour and then stops.

Why location is sensitive. A GPS fix is time plus place plus person in one row. Replay many rows and the path shows home, class, clinic, and prayer hall. That path enables stalking, theft timing, and profiling, so expiry plus narrow share plus revocable consent are the guards.

Other sensitive items work the same way: many people keep social history private between themselves and their physician, so that class of item needs a higher security level than ordinary profile fields.

Worked window math. Live share window $W$ is 0.5 hour or 1 hour. If fixes arrive every 30 seconds, the exposed count is $W \times 120$ fixes per hour. For $W = 0.5$ , that is 60 fixes; for $W = 1$ , 120 fixes. Short window caps the trail. Sense-check: expiry cuts the path even if the receiver saves the chat.

Scope: sensitivity rises with precision and join power. Assumption: coarse city is less risky than exact pin plus time. Bucketing or expiry lowers risk; exact live pins raise it to the top tier.

Recap: exact place plus time identifies and endangers, so share short and stop. Bridge: HIPAA next shows who must guard health-side sensitive items by law.

15.5.3 HIPAA Scope and Covered Parties

HIPAA is the Health Insurance Portability and Accountability Act, and PHI is protected health information, also stated as patient health information.

Who is covered. The regime applies to covered entities: health care providers, health plans, and health care clearinghouses, plus business associates that handle items on their behalf. Large hospitals sit on the covered side while service firms play the business associate role, so both sides of a hospital IT contract inherit protection duties.

Think of a hospital kitchen and its food vendor. The kitchen must keep meals safe; the vendor that carries trays inside must follow the same hygiene rules while inside. Care data works the same: the provider holds the duty, the associate inherits it for the handled slice.

Worked contract split. Hospital holds scan archive. Analytics vendor receives de-identified frames for tumor flagging, returns scores, never holds names. Covered side owns records, associate side guards the slice. Sense-check: the vendor still signs guards even though names never arrive, because re-link risk remains.

Scope: associate duty follows the handled items. Assumption: contract names the slice and the guards. A vague handover with full table access breaks minimization and widens breach scope.

Recap: providers, plans, clearinghouses, plus associates share the duty. Bridge: three rules next say what that duty demands.

15.5.4 The Three HIPAA Rules

Rule triad. Three rules carry the regime. The privacy rule sets the national standard for protecting individually identifiable health information. The security rule sets national standards for protecting the confidentiality, integrity, and availability of electronic PHI. The breach notification rule requires covered entities to notify affected individuals, the health secretary, and potentially the media when a breach occurs. The health secretary here is the HHS secretary named in class. After a midday break the class deliberately reads these aloud so nobody drifts off, which itself signals that examiners love rule triads.

Read them as lock, vault, and alarm. Privacy rule is the lock policy: who may open which file. Security rule is the vault build: guards for electronic PHI across confidentiality, integrity, and availability. Breach notification is the alarm drill: tell affected individuals, tell the health secretary, and bring in media notice when the scale demands it.

Worked alarm path. Lost laptop holds 800 electronic PHI rows unencrypted. Steps: contain, assess scope, notify 800 individuals, notify the health secretary, prepare media notice per scale rule, log fix. Count, tell, fix, log. Sense-check: encryption plus remote wipe would have cut this path to a near miss.

Scope: rules stack, not swap. Assumption: electronic PHI faces all three at once. A team that encrypts but never drills notification still fails the third rule on breach day.

Exam note: write all three with purpose: privacy for identifiable health info, security for electronic PHI across confidentiality, integrity, availability, breach notification to individuals plus health secretary plus media. Bridge: with law set, the course turns to why ML teams must live inside it.

15.5.5 Student Questions and Answers

Q: What are the three key HIPAA rules, and can they be read in detail or only as highlights?

A: Read all three with their purpose. The privacy rule protects individually identifiable health information. The security rule protects electronic PHI across confidentiality, integrity, and availability. The breach notification rule triggers notices to affected individuals, the health secretary, and potentially the media after a breach. The theatre tracking story shows why: a leaked precise location or health fact harms safety, so the lock, vault, and alarm must all hold.

15.6 Why Privacy Rules Belong in Machine Learning Work

15.6.1 The Core Question

Why does a data management for machine learning course teach security, policy, and privacy at all, and where does any of it appear in code, tools, or pipelines?

The class answers through six different voices and then a synthesis. Every answer survives here because each one reaches a different kind of learner: imaging, contracts, daily practice, architecture, minimization, and institutions. Together they show that privacy is not a side chapter but the frame around every ML job that touches protected items.

Answer in one line. Every ML job touches protected items, so protection appears in masking and synthetic stand-ins for training, in contracts signed before any client work, in encryption of items at rest and in transit, in role based access, in audit trails and version logs, in feature elimination that drops identifiers, in consent gates for sensitive analysis, and in privacy by design across the full pipeline from input items to hyperparameters.

Worked map. Take a pathology report pipeline. Mask patient identifier, run analytics to flag tumor presence, send scores back for re-identification at the trusted end. Log each insert and delete with time stamps, version the pickle file with what was detected and when. Mask, score, trace, return. Sense-check: a customer can replay any result to its source row without ever exposing names in the middle.

Recap: privacy lives in code, tools, ETL, lakes, and pipes, not in a PDF. Bridge: six voices next give each learner a home base.

15.6.2 Six Viewpoints From the Discussion

First, a medical imaging viewpoint: models that read scans to find tumors cannot train directly on real patient scans, so the team needs lookalike fictional items that teach the pattern without exposing any real person, or it masks identity so no scan links back to its owner. Government statistic departments and cancer institutes hold similar collections, and the same masking duty applies before any training use. The same caution covers intellectual property: pasting a client name such as a bank into a spreadsheet or report can get the author and the manager sued, so educational providers mask names and keep only the fields the task needs.

Second, a contracts viewpoint: security and compliance form the first contract signed with any client. Health work follows HIPAA, European work follows GDPR, and other regions bring their own regimes. Identity numbers such as social security numbers, Aadhaar numbers, and phone numbers get masked, items in transit and items at rest get encrypted and decrypted under control, and role based access lets only authorized users reach restricted fields.

Encryption plus access in plain words. Items in transit move on wires and need guarded channels. Items at rest sit on disks and need locked stores. Role based access means a role such as radiologist opens scans while billing opens invoices, never the reverse. Masking of social security numbers, Aadhaar numbers, and phone numbers happens before analytics, not after.

Third, a practitioner viewpoint grounded in daily work: skipping protection brings financial, audit, and legal harm. Identification comes first, so a pathology report pipeline masks the patient identifier, runs analytics to flag whether a tumor is present, and sends the report back for re-identification at the trusted end. Security then covers the most sensitive facts such as family history of diabetes, which must never leak into targeting models that emotionally push products at people. Such items open up only with consent, only when the person agrees to that analysis.

Picture a lab with two doors. The inner door holds named reports. The middle room holds numbered slides with no names. Analysts work only in the middle room. Only the trusted desk by the inner door can map numbers back to names.

Fourth, an architect viewpoint: protection shapes documentation, code, tools, ETL jobs, lakes, and pipelines. Security by design reaches the code itself through encapsulation and information hiding. Standards bind everyone equally, and adherence runs at all levels: access control level, logging level, and above. Audit trails record every insertion and deletion on critical items with time stamps, and saved model versions such as pickle files record what was detected and when, so a customer can trace any result back to its origin.

Fifth, a data minimization viewpoint aimed at feature engineering: drop every feature the algorithm does not need. A model that reads four parameters gets those four parameters only, never the patient identifier and never the instrument identifier from the scanner vendor. What is given is decided field by field.

Scope: minimization is per task, not per table. Assumption: the four needed parameters are named and the rest are fenced off. Adding just one identifier for convenience breaks the whole pattern and widens every later risk.

Sixth, an institutional viewpoint: training, ethics instruction, data access guidelines, and privacy by design wrap the whole program. Privacy by design is defined as a set of design principles that fits any digital product, including ML pipelines and ML driven products, and gives a flexible but complete way to hold privacy across every element of ML development. The definition is kept nearly word for word because exam setters favor scenario questions that ask which design principle an ML expert should apply. The answer they want is privacy by design, applied from input items through training and testing to hyperparameter choices.

Worked minimization pass. Model needs four lab values. Input table holds those four plus patient identifier plus scanner vendor identifier plus family history of diabetes. Pass only the four lab values. Four in, three fenced out. Sense-check: accuracy holds while identity, device, and sensitive history never enter training or logs.

Pitfalls. Training on named scans because masking feels slow. Pasting real client names into reports and slides. Logging full rows for debug and forgetting to strip the log. Using family history of diabetes for targeting because the column was already there.

Recap: six voices, one duty — mask first, contract first, encrypt both states, log all, pass least fields, design privacy in. Bridge: methods next give the math that lets models learn while hiding.

Masking and synthetic stand-ins for training, contracts before client work, encryption at rest and in transit, role based access, audit trails with time stamps and pickle version logs, feature elimination, consent gates, and privacy by design from inputs to hyperparameters carry this block into every pipeline.

15.6.3 Student Questions and Answers

Q: Why must we study HIPAA and privacy rules when our subject is data management for machine learning, and where will we ever apply them?

A: Because every ML job touches protected items. Protection appears in masking and synthetic stand-ins for training such as tumor scans and government statistics, in contracts signed before any client work under HIPAA and GDPR, in encryption of items at rest and in transit, in role based access for social security numbers, Aadhaar numbers, and phone numbers, in audit trails and pickle version logs, in feature elimination that drops patient and scanner identifiers down to four needed parameters, in consent gates for sensitive facts such as family history of diabetes, and in privacy by design across the full pipeline from input items to hyperparameters.

Q: A student asks for the privacy by design passage to be read aloud. What does it say?

A: Privacy by design is a set of design principles that can apply to any digital product, including ML pipelines and ML driven products, and it provides a flexible but complete way to ensure privacy in all elements of ML development. Use it as the answer when a scenario asks which design principle an ML expert should apply from inputs through training and testing to hyperparameters.

15.7 Privacy Preserving Machine Learning Techniques

15.7.1 The Four Methods

How can a model learn from hospital or bank records without ever seeing a single named record? Four methods answer that question, each with a different trade.

Differential privacy means the output barely moves when one person opts in or out, so no single row can be read back. Homomorphic encryption means math runs on locked values and only the key holder reads the answer. Multiparty computation means several sites compute together while each keeps its own input hidden. Federated learning means training travels to the data instead of data travelling to one store.

Think of four ways to count cash in sealed envelopes. Differential privacy shakes each count with a little noise so no envelope can be traced. Homomorphic encryption adds the sealed envelopes without opening them. Multiparty splits each note into shares spread across friends so no friend holds a full note. Federation lets each branch count its own till and only the totals meet at head office. Each picture breaks at one point: noise costs accuracy, locked math costs speed, split shares cost coordination, and branch counts still need honest aggregation.

One-line purpose of each. Homomorphic encryption computes on encrypted balances so cloud servers and auditors never see individual amounts. Differential privacy keeps outputs stable when one person leaves the set. Multiparty computation splits trust across sites with per-site signature checks. Federated learning trains in shards such as three shares of 20,000 records out of 60,000 before aggregating on a federal server.

A compact choice table helps for scenario questions:

Method	Hides	Cost	Pick when
Homomorphic encryption	Values during math	High compute	Totals on encrypted balances
Differential privacy	One person's effect	Noise lowers accuracy	Public stats or ML as a service
Multiparty computation	Inputs from each party	Coordination grows with size	Joint check with no trusted hub
Federated learning	Raw rows at center	Sync plus skew	Many sites, one global model

When to pick which in one line: locked math for secret totals, noise for safe release, split shares for joint checks, branches for spread-out training.

Worked scenario sort. Four stems: total assets across banks without showing balances, publish average stay length without leaking one patient, three hospitals score a joint risk list without sharing rows, train one keyboard model across phones without uploading text. Answers in order: homomorphic encryption, differential privacy, multiparty computation with signature checks, federated learning with server aggregation. Match the hiding place to the method. Sense-check: each stem names what must stay hidden and where math must still run.

Exam note: a question may list a scenario and ask which of the four fits, so learn the one-line purpose of each before the machinery: encryption for computing on hidden balances, differential privacy for hiding one person's effect, multiparty for split trust, federated splits for distributed training. Bridge: the next blocks open each machine in the same order.

15.7.2 Homomorphic Encryption

Homomorphic encryption means a cipher that still allows math: combine locked values, unlock once, get the same answer as plain math. A ciphertext is the locked form, a plaintext is the readable form, and key generation issues the public lock plus the private unlock key.

Additive rule. Homomorphic encryption computes over encrypted items so that neither cloud servers nor external auditors see individual balances. The finance setting is exact integer or fixed point transaction math: the system totals assets without revealing any single customer balance. Key generation issues public and private keys, balances are encrypted at entry, and the code never holds a readable balance.

The additive property says that multiplying two ciphertexts yields the encrypted sum of their plaintexts. In symbols, with $E (\cdot)$ the encryption function, $a$ and $b$ plaintext values, and $\oplus$ the ciphertext operation, the rule is:

$E (a) \oplus E (b) = E (a + b)$

Scalar multiplication over ciphertexts is also supported, which covers the weighted sums that ML measures need, since a weight times an encrypted value stays encrypted until the key holder unlocks the total.

Picture two locked cash boxes with a slot that merges them into one locked box holding the combined sum, opened only by the owner key. The axis here is trust: readable values sit only at the two ends with the key holder, locked values fill the whole middle where servers and auditors work. The takeaway in one line: servers add without seeing.

Worked encrypted total. Balances $a = 400$ and $b = 600$ are encrypted as $E (400)$ and $E (600)$ . The server computes $E (400) \oplus E (600) = E (1000)$ without ever holding 400 or 600 in readable form. The key holder unlocks $E (1000)$ to 1000. With a weight $w = 3$ , scalar multiplication gives $3$ times $E (400)$ as $E (1200)$ . 1000 total, zero balances seen. Sense-check: plain math gives $400 + 600 = 1000$ , so locked math matches plain math after unlock.

Credit and debit card rails for international use rely on this family of techniques. The known cost is high computation, so teams reserve it for the narrow steps where secrecy matters most, such as the total step, while plain steps run outside.

Scope: fits narrow exact steps on numbers. Assumption: the needed operations are supported by the chosen scheme and keys stay with the owner. General logic or large model training under full lock stays too slow, so fence the locked zone small.

Pitfalls. Encrypting after the code already logged readable balances. Mixing locked and unlocked values in one sum. Assuming lock equals access control and skipping role checks on who may ask for unlocks.

Recap: locked addition gives secret totals on hidden balances. Bridge: where locked math is too heavy, noise gives the next answer.

15.7.3 Differential Privacy

Differential privacy means a promise in numbers: removing one person barely shifts the released answer. A noisy algorithm adds calibrated random spread to outputs so single rows blur. Machine learning as a service means outsiders send queries and get answers from a model trained on rows they must never read.

Stability promise. Differential privacy keeps the output of a function almost the same whether or not any single person's item is included, which hides the effect of each specific input. In symbols, if $A$ is the output computed with one person's item and $A^{'}$ is the output computed without it, the guarantee is that $A$ and $A^{'}$ stay similar, written as:

$A \approx A^{'}$

where $\approx$ means close enough that one person cannot be detected from the output shift. Training items pass through the noisy algorithm under rules close to role based access control, and the technique shines in third party settings and in machine learning as a service, where items arrive as queries and models must answer without leaking the training set.

Think of a class average with one student stepping out of the room. If the announced average barely moves, nobody learns that student's score. Noise is a light fog over the exact number: shapes stay visible, single faces do not. The picture breaks if the fog is asked about too often, since repeated queries can average the fog away unless a query budget caps them.

Picture a line chart with two nearly overlapping curves, one with the person in and one with the person out. Horizontal axis is the output value, vertical axis is likelihood. The two humps sit almost on top of each other. Takeaway in one line: one row cannot move the hump.

Worked mean shield. Ten ages sum to 300, mean $A = 30$ . Without one member aged 40, the sum is 260 over 9, mean $A^{'} = 28.9$ . Plain release leaks the gap. With noise of plus or minus about 2 added to each release, reported values such as 30.5 and 29.1 overlap, so the leaver blends in. $A \approx A^{'}$ within noise. Sense-check: utility holds for group study while one age stays hidden.

Scope: guards aggregates and trained models, not raw row release. Assumption: noise scale plus query budget are set before release and held. Endless repeat queries or tiny groups break the promise even with noise on.

Recap: noise makes one person's exit invisible. Bridge: where no single hub may hold even noisy inputs, shares give the next answer.

15.7.4 Secure Multiparty Computation

Secure multiparty computation means several parties compute one joint answer while each input stays with its owner. A secret share is one meaningless slice of a split value. A verifier checks every signature from every site before accepting a result.

Multiparty computation refuses to trust a single channel. Three or four components each hold a secret share, each participant contributes securely, and the verifier checks every signature from every site before accepting a result. Actual items are converted into ciphertext before operating, so wires carry locked slices rather than readable rows.

The employment check analogy from class makes it concrete: confirming that a hospital feed truly comes from New Jersey, or confirming that a person truly works somewhere, means asking the supervisor, asking human resources, checking a third party agency, and cross checking professional and bank traces, then accepting only when all sides agree. No single voice decides; agreement across independent voices decides.

Steps in order. Split inputs into shares, spread shares across sites, compute on shares with locked messages, verify every signature from every site, combine only the final answer. The stated drawback is input size: as items and sources grow large, building the joint computation gets hard, since messages and checks multiply with parties and rows.

Picture four desks each holding one corner of a torn map. Nobody can read the route alone. They pass sealed notes, each stamps every note, and only fully stamped notes count toward the joint answer. Takeaway in one line: trust the quorum, not the channel.

Worked quorum check. Hospital feed claims New Jersey origin. Checks: supervisor confirms posting, human resources confirms role, third party agency confirms license, bank trace confirms payroll state. Four yes votes with valid signatures means accept; one missing signature means hold for review. Four of four to pass. Sense-check: a forged feed would need to fool all four independent holders at once.

Scope: fits joint checks and small joint math across owners. Assumption: parties stay online and signatures stay valid through the run. Large tables or many parties push coordination cost past the gain, so sample or pre-aggregate first.

Recap: shares plus every-signature verification replace the trusted hub. Bridge: where data cannot move at all, training moves instead.

15.7.5 Federated Learning

Federated means distributed rather than gathered in one place, because one central store is easy to strike. A federal server aggregates branch updates into one global model without collecting raw rows.

A global model splits across many systems, each site trains or scores its own share, and a federal server aggregates the shares into one result. A worked split from class divides 60,000 records into three shares of 20,000 each, validated separately and then merged. In symbols, with $N = 60000$ total records and $S = 3$ sites, each site holds:

$N / S = 20000$

records per site, so $20000 + 20000 + 20000 = 60000$ in total. Many large vendor systems run this federated pattern across pipelines, algorithms, and data shards so no single breach exposes everything.

Round in order. Server sends current weights down, each site trains locally on its 20,000, each site returns weight shifts only, server averages shifts into new global weights, repeat. Raw rows never leave the branch; only updates travel.

Picture three branch jars each with 20,000 marbles plus one head-office bowl. Each branch sorts its own jar and sends only its sorting notes upward. The bowl merges notes into one recipe. Takeaway in one line: data stays, learning travels.

Worked shard math. Total $N = 60000$ , sites $S = 3$ . Each share is $60000/3 = 20000$ . If site accuracies read 0.90, 0.86, and 0.88, the plain mean is $(0.90 + 0.86 + 0.88) /3 = 0.88$ . Weighted by equal shards the same 0.88 holds. 20,000 each, 0.88 merged. Sense-check: equal shards mean plain and weighted means agree; skewed shards would need weighting by size.

Scope: fits many-site training with similar tasks. Assumption: branches hold enough rows and roughly related mixes. Wildly skewed branches or silent dropouts tilt the global model unless weighting and checks correct for them.

Recap: 60,000 into 3 by 20,000, validated apart, merged once. Bridge: method choice is now testable, as the Q and A locks in.

15.7.6 Student Questions and Answers

Q: Which ML techniques work on items without exposing the underlying set?

A: The four methods are differential privacy, homomorphic encryption, multiparty computation, and federated learning. Encryption hides balances while math still runs with $E (a) \oplus E (b) = E (a + b)$ , differential privacy keeps outputs stable when one person opts out with $A \approx A^{'}$ , multiparty splits trust across three or four sites with supervisor, human resources, third party agency, and bank trace signature checks before accepting a hospital feed from New Jersey, and federation trains in shards of 20,000 out of 60,000 with $N / S = 20000$ before federal server aggregation.

15.8 Anonymization, Pseudonymization, K-Anonymity, and Synthetic Items

15.8.1 Anonymization Basics and Common Tricks

A research team wants real-shaped data with no real person inside. When does a masked table truly stop pointing at anyone?

Anonymization means hiding the identity of the item owner by removing or replacing PII references so no one can be re-identified. Suppression deletes risky characters or cells. Randomization swaps values for plausible but false ones. Masking covers names with invented strings.

Think of a class photo with faces blurred. Blur done well leaves crowd shapes but no face teamwork can restore. Blur done badly is sunglasses that slip off: the key still exists. Only the first counts as anonymized; the second belongs to the keyed family in the next block.

Three working tricks. Suppressing characters removes direct gives such as name tails. Randomizing values shuffles quasi-identifiers so joins fail. Masking names with fake characters replaces each real personal name with an invented string. Many third party firms now sell exactly this service, including PII generation support for test sets. A reversible change is not true anonymization; once the mask can be undone with a key, the result belongs to pseudonymization instead.

Worked mask pass. Row holds name plus PIN code plus visit month. Suppress name to blank, randomize visit month within the quarter, mask phone tail with X characters. Result keeps quarter-level counts for planning while the named owner vanishes. Counts stay, person goes. Sense-check: try to single out one row by joining with a phone book; no join key should survive.

Scope: fits release for reuse and research where no return path is wanted. Assumption: direct plus indirect identifiers are both stripped and joins were tested. Leaving birth date plus PIN code plus gender intact keeps most Americans findable, so test combos, not single columns.

Recap: strip irreversibly or it is not anonymized. Bridge: keyed swaps that allow lawful return come next.

15.8.2 Pseudonymization and Re-identification

Pseudonymization means replacing directly identifying fields such as social security numbers and names with pseudonyms or codes, for example record labels such as ID 1 and ID 2. Re-identification means restoring the real identity through the guarded mapping key. Marshalling and unmarshalling is the Java parallel offered in class, where one transform hides structure for transit and the reverse restores it.

The items stay indirectly identifiable because a holder of the key can re-identify them, which is why the same strict privacy rules still apply. A language analogy from class makes the key concrete: calling someone by an altered nickname hides them from outsiders, while anyone holding the mapping key restores the real name for authorized re-identification. A database masking feature inside SQL Server style engines follows the same mechanism of hiding values in place while the mapping stays guarded.

Three-way comparison to screenshot. Pseudonymized items swap identifiers for pseudonyms and stay indirectly identifiable through a key. Anonymized items irreversibly strip direct and indirect identifiers so no one can re-identify anyone, which frees the set for reuse and research. Synthetic items are generated fresh, discussed next. Students were asked to screenshot this comparison and share it in the class group because exam stems test exactly this split.

Picture three jars. The pseudonym jar holds numbered tokens plus a sealed key envelope on the top shelf. The anonymized jar holds tokens with no envelope anywhere. The synthetic jar holds fresh tokens poured from a generator, never taken from anyone. Takeaway in one line: key on shelf, no key anywhere, never had a key.

Worked code swap. Names map to codes: owner A to ID 1, owner B to ID 2. The analysis table shows only ID 1 and ID 2 with visit facts. The key table mapping ID 1 back to the name sits in a separate locked store with its own access log. Work on IDs, return through key. Sense-check: a leaked analysis table alone names nobody, which is the point of the split.

Scope: fits care and analytics flows that need a lawful return path. Assumption: key and data live apart with separate access and logs. Storing the key beside the coded table collapses the guard to plain text with extra steps.

Recap: codes plus a guarded key allow return; rules still apply in full. Bridge: bucketing next hides even keyless groups.

15.8.3 K-Anonymity With Bucketed Zip Codes

K-anonymity means generalizing quasi-identifiers so each combination appears at least $k$ times. A quasi-identifier is a field that identifies only in combination, such as PIN code plus birth year plus gender. Here $k$ is the minimum group size, a positive integer chosen by the team.

Bucket rule. For a given combination of categories, every bucket must hold at least $k$ records. A worked patient table shows three records sharing generalized values with zip codes 47678, 47673, and 47673 bucketed together so no single row stands alone. The general rule from class: bucket zip codes and similar fields until each bucket holds at least $k$ records, with the bucket width trading utility against safety. Wider buckets mean safer groups but blurrier maps.

Think of school lockers grouped by colour bands instead of numbers. With one locker per band, a note points straight at its owner. With at least $k$ lockers per band, a note points at a crowd. Zip bucketing paints those bands: 47678 with 47673 and 47673 merge into one 4767x band holding three rows.

Picture a bar chart where the horizontal axis is the zip bucket and the vertical axis is row count. Short bars below $k$ are unsafe and must merge. All bars at or above $k$ pass. Takeaway in one line: no short bars allowed.

Worked bucket check. Set $k = 3$ . Three patient rows share generalized birth year plus gender plus bucket 4767x built from 47678, 47673, and 47673. Bucket count is 3, and $3 \geq k$ passes. If one row moved out, the count would read 2 against $k = 3$ and fail, forcing a wider bucket. Three in the bucket meets $k = 3$ . Sense-check: an outsider knowing one patient's zip and year still faces three lookalikes, not one target.

Scope: guards against singling out by known combos. Assumption: the team listed the right quasi-identifiers. Missing one combo field or facing an outside table with finer detail can still isolate a row, so review joins and set $k$ by domain risk such as at least 5 in medical work.

Exam note: state the rule as each combination reaches at least $k$ records, with the worked zip bucket 47678 with 47673 and 47673 as the proof case. Bridge: fresh-made rows give the last option when even buckets are too risky.

15.8.4 Synthetic Items: Uses and Bias Warning

Synthetic items mean generated rows that follow the same patterns as real collections without copying any real person. Generating logic means the code plus assumptions that shape those rows. Responsible AI review means the bias and safety check that sits beside every synthetic feed.

Synthetic items keep work moving when real feeds are missing. A traffic control example generates sensor readings before the true sensors exist, which unblocks sampling and design. Teams can build dashboards, tests, and models on shaped fakes while waiting for live taps.

Generation is random, so a small chance remains that a generated record resembles a real subject, and the deeper flaw is bias: synthetic sets lean whichever way their generating logic leans, through code choices or supporting assumptions. That bias is exactly why responsible AI review sits next to every synthetic feed.

Use plus warning in one place. Use for early builds, load tests, and rare-case drills. Warn that code choices bake lean into every fake row, so downstream models inherit that lean. Review the generator, not only the model, and re-test on real rows once they arrive.

Worked sensor stand-in. Real junction counters arrive in month three. In month one the team generates 10,000 readings from assumed hourly means for sampling design. Dashboards and sampling code ship early. On arrival of real counts, means shift by 15% at peak hours, exposing the assumption lean. Early ship, then recalibrate. Sense-check: fakes unblocked design but never replaced the real calibration run.

Scope: fits design and testing before live data. Assumption: fakes are marked as fakes and bias review is logged. Quietly training a final model on fakes alone and shipping it as real-tested breaks trust and often accuracy.

Recap: synthetic rows unblock early work but carry generator bias. Bridge: Q and A locks the reversible-versus-irreversible split.

15.8.5 Student Questions and Answers

Q: What does anonymization mean, and is masking reversible?

A: Anonymization hides the owner by removing PII references through suppression, randomization, and masking with fake characters. A change that can be reversed with a key is pseudonymization rather than anonymization, and pseudonymized sets keep the full weight of privacy rules. The screenshot comparison holds: pseudonymized stays re-identifiable through ID 1 and ID 2 keys, anonymized never does, synthetic brings bias from generating logic.

Q: How do teams de-identify without losing the ability to re-identify for authorized use?

A: They pseudonymize: swap names and numbers for codes such as ID 1 and ID 2, guard the mapping key apart from the data with marshalling and unmarshalling style transforms for transit, and re-identify only through the key. For research reuse with no re-identification path, they anonymize irreversibly or generate synthetic items such as traffic sensor stand-ins and accept the bias review that comes with them. K-anonymity adds the group guard with zip buckets like 47678 with 47673 and 47673 meeting at least $k$ records.

15.9 Generative AI System Architecture and Security Touchpoints

15.9.1 End to End Flow

One skeleton serves GPT, Gemini, Claude, and DeepSeek alike. Where do safety and audit hooks sit so no stage runs unguarded?

Most generative systems, whether GPT, Gemini, Claude, or DeepSeek, share one skeleton, just as Oracle, Sybase, DB2, Informix, and Postgres share one database skeleton of memory, threads, processes, and control parts. The shared shape is the exam drawing: learn it once, label it for any vendor.

Six stages in order. The flow starts when a user sends a request prompt with an input. The system processes the input and understands intent, and retrieval augmented generation enters at this point. Relevant information comes back through memory and tools. The model generates its output. Post processing runs safety checks and formatting. Finally the response reaches the user as multimodal output: structured items, audio, images, or a mix.

Think of a restaurant line. Order ticket arrives, kitchen reads the ticket and pulls ingredients from stores and tools, chef cooks, pass checks plate safety and dressing, runner serves a tray that may hold a dish plus a drink plus a photo of the special. Retrieval augmented generation is the pantry run between order and cooking: fetch first, then make.

Worked ticket trace. Prompt asks for refund policy plus last invoice total. Retrieval pulls policy page plus invoice row, memory adds prior turn about plan change, generation drafts the reply, post processing checks tone plus number format, response ships as structured invoice plus text. Fetch, draft, check, serve. Sense-check: every claimed number traces to a retrieved row, not to model memory.

Scope: skeleton fits request-driven assistants and retrieval pipelines. Assumption: each stage logs enough to replay. A silent stage that neither logs nor checks breaks both audit and safety at once.

Recap: request, intent plus retrieval, memory and tools, generation, safety post step, multimodal reply. Bridge: entry guards come next.

15.9.2 Entry Layer: Sessions, History, and Rate Limits

Authentication proves who the caller is, authorization proves what they may touch. A session and thread pair tracks one conversation run, while conversation history records what was asked before. Rate limiting caps how often one caller may hit the same API.

The first block after the interface and API gateway is completion session management with context. Authentication and authorization confirm who the caller is, a session and thread track the conversation, and conversation history records what was asked before, the same role cookies and session history played in older web systems. Rate limiting caps how often one caller may hit the same API, enforced through session and threat management with a threshold on every call.

Two stores side by side. Audit counters live beside the main store: actual results and queries go to the vector store with embeddings, while a metadata or admin database records how many times each API was called and what each conversation consumed. Embeddings plus content sit in vectors; counts plus usage sit in admin tables. Mixing them slows both search and audit.

Picture a club door with a counter clicker. The bouncer checks ID, hands a table token for the night, notes each re-entry, and caps entries per hour. The token is the session, the notebook of past visits is history, the hourly cap is the threshold. Takeaway in one line: know who, remember what, limit how often.

Worked threshold math. Threshold allows 60 calls per minute per key. A client sends 90 in one minute. First 60 pass, 30 queue or fail with a retry note, and the admin database logs 90 attempts with 60 served. Cap holds, log shows all 90. Sense-check: audit reads load without touching vector search speed.

Scope: entry guards fit all public APIs. Assumption: clocks and keys are sound. Shared keys or drifting clocks blur who called how often, so issue per-caller keys and sync time first.

Recap: check caller, hold session and thread, remember history, cap by threshold, count in admin store. Bridge: orchestration takes the checked prompt next.

15.9.3 Orchestration, Retrieval, and Memory

Orchestration means agents that split, route, and join work. Retrieval searches vector stores, keyword indexes, or hybrid combinations. Short term memory stages the working set for the current reply.

After entry checks, orchestration takes over through agents. A coordinator fans work out to task agents that may read text, audio, or other channels, analyze the pieces, and settle intent. Retrieval then searches vector stores, keyword indexes, or hybrid combinations for the available material, calls tools, and stages the result in short term memory or a database. A response planner drafts the reply using previous and related responses, which is where graph stores earn their keep. Model building selects the right model, transformer, and encoder, predicts the next token, and tunes with the chosen optimization. Training feeds arrive offline from synthetic, public, and licensed sources. Security, confidentiality, auditing, and governance start at the entry layer and repeat at the vector store, the knowledge graph, and every API.

Governance repeats at every store. Security, confidentiality, auditing, and governance start at the entry layer and repeat at the vector store, the knowledge graph, and every API. No hop is trusted by position; each hop checks and logs again.

Picture a newsroom desk. Coordinator splits a story across beat reporters for text, audio, and records. Library pulls clips by meaning plus keywords. Desk holds the day file in short memory. Editor drafts from prior plus related pieces. Standards desk re-checks at each handoff. Takeaway in one line: many hands, one standards sheet.

Worked plan trace. User asks for plan comparison with prior bill. Coordinator tasks one agent for bill rows and one for plan pages. Retrieval returns top vector hits plus keyword invoice IDs. Planner drafts from previous plus related responses stored as graph links. Split, fetch, link, draft. Sense-check: the graph hop is what keeps follow-ups coherent across turns.

Scope: multi-agent retrieval fits linked multi-turn tasks. Assumption: stores hold fresh indexed material. Stale vectors or missing keyword fields push the planner to invent, so index freshness is a release gate.

Recap: coordinate, retrieve hybrid, stage short memory, plan from linked prior work, govern each hop. Bridge: store choice makes this fast.

15.9.4 Choosing the Store

Graph stores such as Neo4j hold linked previous and related responses where links are first class. NoSQL options such as DynamoDB and Cassandra hold flat high scale admin facts. GraphQL based access is the query style named alongside. Vector stores hold embeddings for similarity search.

Match store to pattern. Teams often pick graph stores such as Neo4j for flexibility because previous and related responses link naturally as graphs. Alternatives named include DynamoDB and Cassandra on the NoSQL side and GraphQL based access, alongside vector graph options for linked plus similarity needs. One student runs Neo4j with certification level comfort, and another student applied graph queries in their own build. The practical rule is to match the store to the access pattern: vectors for similarity, graphs for linked responses, and a separate metadata database for counts and audit.

A short decision table:

Need	Store	Why
Similar chunks	Vector store	Meaning search over embeddings
Linked turns	Neo4j graph	Prior plus related as edges
Flat counts	DynamoDB or Cassandra	High scale key reads
Shaped reads	GraphQL based access	Ask for exactly the fields

When to pick which in one line: similarity goes vector, links go graph, counts go NoSQL, shaped fetches go GraphQL.

Worked placement. Chat app with 2 million turns plus billing counts. Turn texts plus embeddings go vector, reply links go Neo4j, per-key call counts go DynamoDB or Cassandra admin tables. Three patterns, three homes. Sense-check: audit never scans vectors and search never scans count tables.

Production stacks run this whole shape on compute, storage, database, and network layers with Kubernetes style runtimes and CI CD plus MLOps pipelines underneath.

Scope: polyglot stores win when access splits cleanly. Assumption: writers keep all three in sync. A turn stored in vectors but missing in graph breaks follow-ups; a call missing in admin breaks billing.

Recap: vectors for likeness, Neo4j graphs for links, DynamoDB or Cassandra for flat scale. Bridge: Q and A pins the async plus limit plus store splits.

15.9.5 Student Questions and Answers

Q: How are asynchronous prompts managed after the UI and API gateway, and where does orchestration begin?

A: The prompt lands in session management with context, then orchestration starts. An agent layer takes the prompt, keeps session and thread state, and routes tasks across text and audio task agents while async calls resolve. History, thresholds, and limits are tracked per call so replays stay consistent, and governance repeats from entry through vector store, knowledge graph, and every API.

Q: Rate limiting is confusing when the same REST API is called many times. How is it enforced and audited?

A: Session and threat management hold a threshold for every call and level the load. Counts and usage land in a metadata or admin database rather than the vector store, so auditing reads who called what and how often without disturbing embeddings. Completion session management with context plus authentication and authorization sits in front, so caps apply per known caller.

Q: Should conversation state sit in a vector database, a JSON file, or a NoSQL store such as DynamoDB or Cassandra, and where does the graph fit?

A: Query results and embeddings sit in the vector store. Counts, audit trails, and admin facts sit in a metadata database. Graph stores such as Neo4j fit linked previous and related responses because links are first class, while DynamoDB or Cassandra fit flat high scale admin facts and GraphQL based access shapes reads. Match the store to the pattern, running the stack on compute, storage, database, and network layers with Kubernetes style runtimes and CI CD plus MLOps underneath.

15.10 Exam Preparation: Syllabus Map and Answer Writing

15.10.1 Pipelines, Profiling, and Quality Dimensions

The paper rewards students who draw the pipe, name the layer, and prove the check. What opening trio earns those marks?

Exam coverage opens with data representation, data architecture, and data pipelines: collection, ingestion, features, and the special case of a pipeline for LLM work, then the infrastructure that hosts them and the ML lifecycle around them. Data profiling and validation ask how a profile gets built for a new customer and how it is checked.

Quality trio. The quality lens from the first weeks returns here: a profile must be accurate, relevant, and complete, because nobody accepts an incomplete customer profile. A healthcare scenario and a retail scenario test whether the right quality dimensions were chosen for the domain: accuracy plus completeness weigh heaviest in care, relevance plus freshness weigh heaviest in retail offers. Validation questions then ask how a built pipeline, whether ETL, data, or LLM, gets proven correct and which validation paths fit which failure types.

Think of opening a bank account for a new customer. Collection gathers ID plus address, profiling summarizes what arrived, validation checks it against sources, and quality gates ask if the file is right, on-topic, and whole before any model reads it.

Worked profile check. New customer file holds name, city, and last three orders but no contact channel. Accurate yes, relevant yes, complete no. Verdict: hold for contact capture before scoring. Two of three is a fail. Sense-check: an incomplete profile silently biases every downstream step, so completeness gates release.

Scope: profiling questions want method plus check. Assumption: state source, rule, and fix per dimension. Naming dimensions without the check path earns only half marks.

Recap: accurate, relevant, complete, proven per pipeline type. Bridge: storage and streaming layers come next.

15.10.2 Architectures, Ingestion, and Metadata

Medallion means bronze for raw inflow, silver for cleaned joins, gold for trusted serving sets. A lake holds raw variety, a warehouse holds structured history, a mart serves one team. Apache Kafka carries streaming events, Apache Spark and Apache Flink process them.

Layer plus stream plus catalog. Storage questions target lake, warehouse, and mart designs through the medallion pattern of bronze, silver, and gold layers: what each layer holds and how raw inflow refines into trusted output. Ingestion questions emphasize streaming with Apache Kafka plus processing with Apache Spark and Apache Flink, flagged with extra weight on the middle lecture block covering ingestion. Metadata questions ask for types of metadata and for ways to build a metadata system, from spreadsheets through cloud repositories to a registry.

Experimentation questions cover train and test splits, evaluation habits, and methods such as k-fold, where $k$ names the number of folds, a positive integer, and each fold takes its turn as the held out test share. With $k = 5$ , each fold holds $1/5$ of rows as the test share in its turn, so every row is tested once across five rounds. With the middle block and the metadata block carrying extra weight, diagrams of layers plus one line per layer score well.

Worked k-fold split. Rows $N = 1000$ , folds $k = 5$ . Each test share holds $1000/5 = 200$ rows. Round one tests rows 1 to 200 while training on 800, round two tests 201 to 400, and so on through five rounds. 200 tested each round, all 1000 tested once. Sense-check: matches the held-out-share rule with no row always hidden and none always tested.

Scope: ingestion and metadata carry extra weight, so budget time there. Assumption: draw plus label plus one line of duty per layer. A bare box diagram without the duty line misses the architecture mark.

Recap: bronze, silver, gold plus Kafka, Spark, Flink plus catalog from sheet to registry plus k-fold by turns. Bridge: compute and pipeline math come next.

15.10.3 LLM Internals, Big Data, and Compute Choices

MapReduce splits work into map steps per block plus a reduce merge, with HDFS holding the blocks. Symmetric multiprocessing shares one memory across processors, clusters give each node its own. Shared nothing scales by adding nodes, shared everything shares disks or memory. A directed acyclic graph, shortened to DAG, orders pipeline stages with no loops.

LLM questions ask for the parts of a model and how a model gets built for a scenario, including components and component choices. Big data questions revisit MapReduce and HDFS, sometimes as a distribution exercise: given inputs and four nodes, show how blocks split, where map runs, and how reduce merges.

DAG in symbols. Pipeline automation questions ask for the directed acyclic graph, the DAG, where each pipeline stage is a node and each dependency is an edge. In symbols, $G = (V, E)$ with $V$ the set of stage nodes and $E$ the set of dependency edges, which must never form a cycle. Ordered stage dependencies such as pipeline one before pipeline two become edges from the first node to the next.

Compute questions contrast parallel against distributed designs: two processors sharing one memory versus every processor holding its own memory, symmetric multiprocessing against clusters, and shared nothing versus shared everything for a cost sensitive bank scenario. Service level agreement questions ask how an SLA gets built for a pipeline so latency, freshness, and duty boundaries hold.

Worked four-node split. Input of 8 blocks on four nodes places 2 blocks per node. Map runs locally on all four at once, shuffle groups by key, reduce merges into one sorted output. DAG for this reads $V =$ {split, map, shuffle, reduce} with $E =$ {split to map, map to shuffle, shuffle to reduce} and no cycle. 2 blocks each, one merge. Sense-check: moving compute to blocks beats moving blocks to compute, which is the HDFS plus MapReduce point.

Scope: draw the split plus the DAG side by side. Assumption: label nodes as stages and edges as must-finish-before links. A cycle in the drawing fails the DAG definition on sight.

Production picks follow cost: shared nothing clusters for bank scale-out, shared memory only where tight coupling pays.

Recap: blocks split, map local, reduce merges; $G = (V, E)$ with no cycle. Bridge: privacy and answer craft close the map.

15.10.4 Privacy, Explainability, and Audit on the Paper

The privacy block from this session and the prior one contributes term questions: data anonymity, privacy itself, explainability, and the auditing each one demands. Trusted answers link terms to actions: anonymized fields, consent gates, logged pipeline stages, populated metadata with time stamps, and named techniques such as differential privacy or federated splits where the scenario fits.

Medium-first method. Scenario stems may be long, for example asking how a retrieval pipeline should be re-architected to remove tabular hallucination without overloading context, then dropping the demand to a medium analysis level. When a stem feels heavy, answer the medium version first: name the failure, name the fix, name the check. For the table case that reads: hallucinated cells from memory, limit context to needed rows with per-cell retrieval grounding, check rendered cells against retrieved rows.

Worked table fix. Table question needs 6 rows of 200. Pull only those 6 by key, ground each cell to its source row, render, then diff rendered cells against pulls before answering. 6 rows in, 6 cells checked. Sense-check: narrow context plus cell checks beat stuffing 200 rows into the window.

Scope: term answers need action plus audit. Assumption: each named technique pairs with its log line. Naming differential privacy without the noise-plus-budget line, or federation without the shard line, reads as a label without proof.

Recap: term to action to log line, medium version first. Bridge: scoring habits make these answers land.

15.10.5 How to Write Answers That Score

Presentation advice is blunt. Papers are not corrected by the person teaching, assistants share the load, and in this course a single evaluator may read the whole script, so the first answers set the impression for everything after. Read each question two or three times before writing. Draw the architecture diagram. Answer in bullets and numbered steps. Underline the important line. Never paste long copied passages. Stay relevant to the stem and keep every answer complete from the early questions onward.

Study method advice points at active recall over rereading. Upload the notes PDF into a study tool such as NotebookLM, ask it for the architecture alone, drill the diagrams with generated questions, generate audio for revision, and build mind maps such as a compliance map whose pillars read consent, transparency, minimization, security, accountability, and user rights. Take one diagram at a time with its explanation, then close the file and redraw it from memory.

Scoring checklist. Read the stem two or three times, sketch the diagram first, write in bullets and numbered steps, underline the key line, stay on the stem, finish early answers fully since they set the impression for the whole script.

Worked first-answer plan. Ten-mark architecture stem: two minutes reading twice, three minutes sketching medallion plus pipe plus stores, five minutes in six bullets with one underlined verdict line. Diagram plus bullets plus one underline. Sense-check: an evaluator scanning in seconds still sees structure, labels, and verdict.

Scope: craft multiplies knowledge; it never replaces it. Assumption: content lines are right first. Neat wrong answers still score low, so verify theMapReduce split, the DAG edges, and the pillar names before inking the underline.

Recap: read twice, draw, bullet, underline, stay relevant, finish strong early. Bridge: Q and A shows the method on a hard stem.

15.10.6 Student Questions and Answers

Q: How should a retrieval pipeline be re-architected to remove tabular hallucination without overloading context?

A: Treat it as a medium analysis task. Name where the table facts enter, limit the context to the rows the question needs such as 6 of 200, ground each cell through retrieval rather than memory with MapReduce and HDFS style block discipline behind the stores, and check the rendered table against the retrieved rows before answering. Hold the DAG shape $G = (V, E)$ across stages and test by k-fold style turns where $k$ sets the held out share, so tabular answers stay grounded without context overload.

Exam Guidance Summary

Feature engineering appears in every project type: ML, deep learning, and LLM. Only pass the fields the step needs and drop identifiers early, such as four lab values in with patient and scanner identifiers fenced out.
Temperature control is examinable: low values such as $T = 0$ , $T = 0.1$ , and $T = 0.2$ for deterministic factual work including Java code, C++ code, basic mathematics, factual question answering, technical documentation, and the jet engine explainer at $T = 0.1$ ; medium warm for semi-structured writing such as emails and blogs; high novel and creative for stories.
The six compliance pillars answer perspective questions: accountability, subject rights, consent, transparency, data minimization, and security, shown in the shared drive grant of one transformer code file to one classmate with read access only.
Memorize the ten GDPR requirements in order — limitation of purpose, data, and storage; consent; privacy by design; data transfer into lake, warehouse, or hub; awareness training about every six months; data protection officer; data protection impact assessment; personal data breaches; data subject rights; lawful, fair, and transparent processing — and the three HIPAA rules with one line of purpose each: privacy for identifiable health info, security for electronic PHI across confidentiality, integrity, availability, breach notification to individuals plus health secretary plus media.
Tie each label to its regime: PII to general identity practice, SPI to the California category with geolocation and health examples such as longitude, latitude, and GPS pins, PHI to HIPAA with social history such as drink-or-smoke notes as the worked case.
India's Digital Personal Data Protection Act carries a stated minimum penalty of 50 crore rupees with count $50$ , a grievance redress right, child well-being limits, and narrow exemptions for legal claims before a tribunal or court and specific investigations.
Privacy by design is the expected answer for scenario questions asking which principle an ML expert applies across pipelines and products, from input items through training and testing to hyperparameters.
The four ML techniques map to scenarios: homomorphic encryption for computing on encrypted balances with $E (a) \oplus E (b) = E (a + b)$ , differential privacy with $A \approx A^{'}$ for hiding one person's effect in third party and ML as a service settings, multiparty computation with per-site signature checks across supervisor, human resources, third party agency, and bank traces for split trust such as a New Jersey hospital feed, federated learning with the 60,000 into 3 by 20,000 split with $N / S = 20000$ for distributed training.
The triple comparison is screenshot worthy: pseudonymized items stay re-identifiable through a key such as ID 1 and ID 2 with marshalling style return, anonymized items never do, synthetic items such as traffic sensor stand-ins bring bias from their generating logic.
K-anonymity buckets quasi-identifiers until each combination reaches at least $k$ records; the worked zip bucket is 47678 with 47673 and 47673 meeting $k = 3$ .
Draw the generative system skeleton end to end: request, session and thread with history, retrieval with memory and tools, generation, safety post processing, multimodal response, with governance at entry, vector store, knowledge graph, and every API; place vectors for similarity, Neo4j graphs for linked responses, DynamoDB or Cassandra for flat counts, GraphQL for shaped reads, on Kubernetes style runtimes with CI CD plus MLOps.
Weighted syllabus zones: ingestion architectures with Apache Kafka, Apache Spark, and Apache Flink; medallion bronze, silver, gold; metadata systems from sheets to registry; experimentation with k-fold where $k$ sets the held out share; MapReduce with HDFS on four nodes; parallel versus distributed with symmetric multiprocessing versus clusters and shared nothing versus shared everything; DAG as $G = (V, E)$ ; SLA construction; LLM components with SNOMED CT grounding and LangChain plus Streamlit plus pandas plus SHAP plus XGBoost tooling.
Write to score: read each stem two or three times, draw the diagram, use bullets and numbers, underline key lines, stay relevant, keep the first answers strong since one evaluator may read the whole script, and drill with NotebookLM plus mind maps plus redraw-from-memory practice.

Key Industry Applications

UPI scale payment rails use LLM plus LangChain chains for zero-shot failure categorization into 15 classes with 5 Whys, Ishikawa diagrams, pandas frames, and a Streamlit dashboard of 8 views backed by logistic regression, random forest, and XGBoost baselines with SHAP checks, triaging about 10.8 million daily failures from a $3%$ rate.
Hospital data pipelines mask patient identifiers, run pathology analytics on de-identified frames, and re-identify only at the trusted end for tumor flagging, with audit trails plus pickle version logs and role based access throughout.
Card rails for international payments compute over encrypted balances with homomorphic encryption under $E (a) \oplus E (b) = E (a + b)$ so cloud servers and auditors never see individual amounts.
Employment and hospital feed verification runs multiparty style checks across supervisors, human resources, third party agencies, and bank traces with every-signature verification before acceptance, the New Jersey feed pattern.
Large vendor estates train federated global models across shards such as three shares of 20,000 records out of 60,000 with $N / S = 20000$ and federal server side aggregation so no single breach exposes everything.
Chat location sharing grants half hour to one hour windows because precise GPS, longitude, and latitude count as sensitive personal information that can reveal live position.
Graph stores such as Neo4j hold linked conversational responses while DynamoDB or Cassandra style NoSQL holds flat high scale admin facts with GraphQL shaped reads, on compute, storage, database, and network layers with Kubernetes runtimes and CI CD plus MLOps underneath.
Clinical grounding pairs SNOMED CT concept identifiers with property knowledge graphs and fused LLMs plus orchestrators so predictions stay tied to curated truth where a wrong answer harms patients.

DMML Lecture 15 notes · Privacy Governance and Responsible Machine Learning

Data Management for Machine Learning· postgraduate· 2026-09-15

Sections Breakdown

1Student Project Showcase: Payment Failures and Medical Knowledge Graphs

UPI payment failure RCA pipeline with LangChain zero-shot categorization into 15 classes, 5 Whys plus Ishikawa review, pandas flow, Streamlit dashboard, model comparison won by Claude Sonnet on JSON discipline, SHAP checks, and SNOMED CT grounding in the medical build.

2LLM Selection, Prompting, and the Temperature Control

Temperature control bands: low deterministic most-likely completion for code math facts docs, warm middle for semi-structured writing, high novel creative end, with jet engine task pinned at T=0.1.

3Data Privacy Foundations: Breach, PII, and Compliance Pillars

Breach to privacy to compliance chain, PII as Personal Identifiable Information finder step, six pillars with shared-drive single-file read-only grant and doctor-patient-hospital rights split.

4Legal Frameworks: GDPR, the Indian Act, and Protection Practice

GDPR ten requirements mapped to lake warehouse hub work, seven lawful-processing duties, Indian Digital Personal Data Protection Act penalty rights and tribunal exemptions, policy-to-SLA practice, DPIA change-control gate.

5Sensitive Information: SPI, PHI, and HIPAA Rules

PII versus SPI California category versus PHI health law, GPS theatre misuse case with half-hour to one-hour windows, HIPAA covered entities plus associates, privacy security breach-notification triad.

6Why Privacy Rules Belong in Machine Learning Work

Six viewpoints for why ML work needs privacy: imaging masking, contracts first, practitioner pathology masking plus consent, architect audit trails, four-parameter minimization, privacy by design institutions.

7Privacy Preserving Machine Learning Techniques

Four privacy preserving methods with choice table: homomorphic locked addition plus scalar weights, differential A approx A-prime noise, multiparty shares with supervisor HR agency bank quorum, federated 60000 into 3 by 20000 with federal aggregation.

8Anonymization, Pseudonymization, K-Anonymity, and Synthetic Items

Suppression randomization masking trio, ID 1 ID 2 pseudonym codes with guarded key and marshalling parallel, k-anonymity zip bucket 47678 with 47673 and 47673, synthetic traffic sensor stand-ins with generator bias review.

9Generative AI System Architecture and Security Touchpoints

Generative AI skeleton across GPT Gemini Claude DeepSeek: request intent retrieval memory tools generation safety multimodal reply, session thread history thresholds, hybrid orchestration, Neo4j versus DynamoDB Cassandra GraphQL store choice on Kubernetes MLOps base.

10Exam Preparation: Syllabus Map and Answer Writing

Syllabus map: pipelines profiling quality trio, medallion plus Kafka Spark Flink plus registry with k-fold turns, MapReduce HDFS four-node split plus G=(V,E) DAG plus shared nothing versus everything, tabular anti-hallucination method, read-twice draw bullet underline scoring.

11Exam Guidance Summary

Exam guidance appendix carried through with temperature bands, pillar names, GDPR order, HIPAA triad, technique map, k-anonymity bucket, skeleton sketch, weighted zones, scoring habits.

12Key Industry Applications

Industry applications appendix carried through: UPI RCA, hospital masking, encrypted card rails, quorum checks, federated shards, location windows, polyglot stores, SNOMED grounding.

Postgraduate students studying data management for machine learning

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

Student Project Showcase: Payment Failures and Medical Knowledge Graphs

Must-know: UPI 3% rate means about 10.8M daily failures sorted into 15 classes with 5 Whys, Ishikawa, pandas, 8-view Streamlit dashboard

$F = N \times r$

⚠️ Top pitfall: Treating the LLM label as ground truth without sampling against logs

Self-check: Why was Claude Sonnet picked over cheaper models?

Connects to: Section 15.2 — LLM Selection, Prompting, and the Temperature Control

LLM Selection, Prompting, and the Temperature Control

Must-know: Low T=0/0.1/0.2 for exact work including jet engine explainer, medium warm for emails and blogs, high for novels

$T = 0.1$

⚠️ Top pitfall: Leaving temperature high for extraction and JSON output

Self-check: Which band fits a factual jet engine explanation?

Connects to: Section 15.1 — Student Project Showcase: Payment Failures and Medical Knowledge Graphs

Data Privacy Foundations: Breach, PII, and Compliance Pillars

Must-know: Breach prompts privacy which demands compliance; PII is the finder step; six pillars are accountability subject rights consent transparency minimization security

⚠️ Top pitfall: Holding fields with no stated purpose or guard

Self-check: What are the six compliance pillars?

Connects to: Section 15.4 — Legal Frameworks: GDPR, the Indian Act, and Protection Practice

Legal Frameworks: GDPR, the Indian Act, and Protection Practice

Must-know: Ten GDPR requirements in class order; lawful limited small accurate short-lived guarded owned; Indian Act 50 crore floor with grievance right and narrow lanes

$50 \times 1 0^{7}$

⚠️ Top pitfall: Reusing data for a second purpose without its own basis

Self-check: Name the ten GDPR requirements in order.

Connects to: Section 15.3 — Data Privacy Foundations: Breach, PII, and Compliance Pillars, Section 15.5 — Sensitive Information: SPI, PHI, and HIPAA Rules

Sensitive Information: SPI, PHI, and HIPAA Rules

Must-know: PII names, SPI stings, PHI heals under lock; HIPAA lock vault alarm triad with health secretary and media notice

⚠️ Top pitfall: Mixing PII SPI PHI labels and their regimes

Self-check: What are the three HIPAA rules with one-line purposes?

Connects to: Section 15.3 — Data Privacy Foundations: Breach, PII, and Compliance Pillars, Section 15.4 — Legal Frameworks: GDPR, the Indian Act, and Protection Practice, Section 15.6 — Why Privacy Rules Belong in Machine Learning Work

Why Privacy Rules Belong in Machine Learning Work

Must-know: Six voices one duty: mask first contract first encrypt both states log all pass least fields design privacy in; privacy by design is the scenario answer

⚠️ Top pitfall: Training on named scans or pasting client names into reports

Self-check: Which design principle applies privacy across ML pipelines and products?

Connects to: Section 15.3 — Data Privacy Foundations: Breach, PII, and Compliance Pillars, Section 15.7 — Privacy Preserving Machine Learning Techniques, Section 15.8 — Anonymization, Pseudonymization, K-Anonymity, and Synthetic Items

Privacy Preserving Machine Learning Techniques

Must-know: Four methods to scenarios: locked totals, A approx A-prime noise, every-signature quorum, 60000 into 3 by 20000 federation

$E (a) \oplus E (b) = E (a + b)$

⚠️ Top pitfall: Using locked math for bulk training or endless queries against noisy release

Self-check: Which method totals bank balances without seeing any single balance?

Connects to: Section 15.6 — Why Privacy Rules Belong in Machine Learning Work, Section 15.8 — Anonymization, Pseudonymization, K-Anonymity, and Synthetic Items

Anonymization, Pseudonymization, K-Anonymity, and Synthetic Items

Must-know: Key on shelf versus no key anywhere versus fresh pour; each zip bucket reaches at least k with 47678/47673/47673 proof

$k$

⚠️ Top pitfall: Storing the mapping key beside the coded table

Self-check: What distinguishes pseudonymized from anonymized from synthetic?

Connects to: Section 15.7 — Privacy Preserving Machine Learning Techniques

Generative AI System Architecture and Security Touchpoints

Must-know: Six-stage skeleton with governance at every hop; vectors for likeness Neo4j for links NoSQL for counts

⚠️ Top pitfall: Storing counts in vectors or embeddings in admin tables

Self-check: Where do embeddings sit versus call counts?

Connects to: Section 15.10 — Exam Preparation: Syllabus Map and Answer Writing

Exam Preparation: Syllabus Map and Answer Writing

Must-know: Accurate relevant complete; bronze silver gold plus Kafka Spark Flink plus k-fold turns; G=(V,E) no cycle; read twice draw bullet underline

$G = (V, E)$

⚠️ Top pitfall: Neat wrong answers: verify splits DAG edges pillar names before underlining

Self-check: What is the DAG in symbols and what is forbidden in it?

Connects to: Section 15.9 — Generative AI System Architecture and Security Touchpoints

Previous LectureLecture 14 · Data Privacy, Governance and LLM Data Pipelines

Next LectureLecture 16 · Machine Learning Lifecycle, Data Pipelines, and Deployment

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Privacy Governance and Responsible Machine Learning

15.1 Student Project Showcase: Payment Failures and Medical Knowledge Graphs

15.1.1 Root Cause Analysis of UPI Payment Failures With LangChain and LLM

15.1.2 Failure Categories, Feature Engineering, and Severity Rules

15.1.3 Model Comparison, Prompt Design, and Explainability

15.1.4 Medical Grounding With SNOMED Concepts and a Second Project

15.1.5 Student Questions and Answers

15.2 LLM Selection, Prompting, and the Temperature Control

15.2.1 Why Temperature Matters in Prompting

15.2.2 Worked Guide: Which Band for Which Job

15.2.3 Student Questions and Answers

15.3 Data Privacy Foundations: Breach, PII, and Compliance Pillars

15.3.1 Breach, Information Privacy, and Why Businesses Collect Data

15.3.2 Personal Identifiable Information

15.3.3 The Six Pillars That Shape Every Privacy Question

15.3.4 Student Questions and Answers

15.4 Legal Frameworks: GDPR, the Indian Act, and Protection Practice

15.4.1 General Data Protection Regulation and Its Ten Requirements

15.4.2 Lawful Processing Principles

15.4.3 The Indian Digital Personal Data Protection Act

15.4.4 From Policy to Contract: Making Protection Real

15.4.5 Impact Assessment Borrowed From Change Management

15.4.6 Student Questions and Answers

15.5 Sensitive Information: SPI, PHI, and HIPAA Rules

15.5.1 PII Versus SPI Versus PHI

15.5.2 What Counts as Sensitive

15.5.3 HIPAA Scope and Covered Parties

15.5.4 The Three HIPAA Rules

15.5.5 Student Questions and Answers

15.6 Why Privacy Rules Belong in Machine Learning Work

15.6.1 The Core Question

15.6.2 Six Viewpoints From the Discussion

15.6.3 Student Questions and Answers

15.7 Privacy Preserving Machine Learning Techniques

15.7.1 The Four Methods

15.7.2 Homomorphic Encryption

15.7.3 Differential Privacy

15.7.4 Secure Multiparty Computation

15.7.5 Federated Learning

15.7.6 Student Questions and Answers

15.8 Anonymization, Pseudonymization, K-Anonymity, and Synthetic Items

15.8.1 Anonymization Basics and Common Tricks

15.8.2 Pseudonymization and Re-identification

15.8.3 K-Anonymity With Bucketed Zip Codes

15.8.4 Synthetic Items: Uses and Bias Warning

15.8.5 Student Questions and Answers

15.9 Generative AI System Architecture and Security Touchpoints

15.9.1 End to End Flow

15.9.2 Entry Layer: Sessions, History, and Rate Limits

15.9.3 Orchestration, Retrieval, and Memory

15.9.4 Choosing the Store

15.9.5 Student Questions and Answers

15.10 Exam Preparation: Syllabus Map and Answer Writing

15.10.1 Pipelines, Profiling, and Quality Dimensions

15.10.2 Architectures, Ingestion, and Metadata

15.10.3 LLM Internals, Big Data, and Compute Choices

15.10.4 Privacy, Explainability, and Audit on the Paper

15.10.5 How to Write Answers That Score

15.10.6 Student Questions and Answers

Exam Guidance Summary

Key Industry Applications

DMML Lecture 15 notes · Privacy Governance and Responsible Machine Learning

Sections Breakdown

Exam Revision Notes

Student Project Showcase: Payment Failures and Medical Knowledge Graphs

LLM Selection, Prompting, and the Temperature Control

Data Privacy Foundations: Breach, PII, and Compliance Pillars

Legal Frameworks: GDPR, the Indian Act, and Protection Practice

Sensitive Information: SPI, PHI, and HIPAA Rules

Why Privacy Rules Belong in Machine Learning Work

Privacy Preserving Machine Learning Techniques

Anonymization, Pseudonymization, K-Anonymity, and Synthetic Items

Generative AI System Architecture and Security Touchpoints

Exam Preparation: Syllabus Map and Answer Writing

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat