Adversarial Machine Learning Fundamentals

14.1 Anatomy of a Machine Learning System

A machine learning system is far more than a single model file. It is a chain of moving parts that starts with a problem statement and ends with a decision in production. An attacker can target any link in that chain. That is why the first step is to name every part with care.

Think of the whole system as a water supply line running from a reservoir to a tap. The model is only the filter in the middle. Poison the reservoir (training data), crack a pipe joint (feature logic), misread the meter (evaluation), or swap the tap (deployment), and clean water still seems to flow until someone gets sick. Each later section of this lecture attacks one stretch of that pipe.

14.1.1 From Problem Framing to Deployment

Work starts with problem framing. A team decides what to predict. A malware team might decide to predict whether a Windows binary is benign or malicious. A network team might decide to predict whether a flow is part of a denial-of-service flood. That decision shapes everything that follows, because it fixes the label set, the cost of each error type, and the response that each verdict triggers.

Next comes data collection. The team gathers raw material. For malware this can be binaries, sandbox logs, API call traces, header fields, strings, and entropy signals. For network work this can be packet captures, flow records, login times, and byte counts. The collection step also includes labeling. Someone must mark which items are benign and which are malicious. Labeling is slow and skilled work. Trained students and analysts often spend days on it. Errors here spread everywhere downstream, which is why label poisoning in a later section is so damaging: one edited label store quietly retrains every future model.

Then comes preprocessing and feature work. Raw data is cleaned. Duplicates are removed. Missing values are handled. Useful signals are turned into numbers a model can read. Examples are average packet size, number of crypto API calls, section entropy of a binary, login hour, and count of failed logins. Some teams store these in a feature store — a shared place where the same feature logic serves both training and live scoring. The point of a feature store is to keep training and serving consistent. If the two sides compute a feature in different ways, the model will stumble in production even when test scores look strong. A classic failure is training on one parsing of a binary header while the live agent parses the same header a different way, so the feature the model learned never arrives at scoring time.

Training follows. The team picks a model family, sets hyperparameters, handles class imbalance, and fits weights on the labeled data. In security work benign items vastly outnumber malicious ones. Most traffic is benign. Most files are benign. One missed attack can still be very costly. So the team must think about recall on the rare class, not just overall accuracy. Techniques include class weights, resampling, threshold tuning, and careful split design. The threshold deserves attention now because evasion later is defined against it: the team picks a score cut-off $\tau$, and every file scoring at or above $\tau$ is treated as malicious.

Evaluation comes next. The team measures accuracy, precision, recall, false positive rate, and related scores on held-out data. A good team also slices results by malware family or attack type. A single global number can hide a blind spot on one family. Precision and recall make the trade-off exact. Write $TP$ for true positives, $TN$ for true negatives, $FP$ for false positives, and $FN$ for false negatives. Then precision and recall are:

$$\text{Precision}=\frac{TP}{TP+FP}$$

$$\text{Recall}=\frac{TP}{TP+FN}$$

Here precision answers how many flagged files were truly malicious, while recall answers how many malicious files were caught. A security team usually protects recall on the rare class even at the price of extra analyst triage from lower precision.

Deployment wraps the model in software. The model joins an inference path with input parsing, feature extraction, model scoring, threshold logic, logging, and response actions such as block, quarantine, or alert. Tools receive the score and act on it.

A malware detector that lives inside an endpoint agent parses each new binary, extracts entropy and API-call features, scores the file, and tells the agent whether to allow or quarantine it. The same shape holds for a network sensor: parse each flow, build timing and size features, score, and either allow, throttle, or raise an alert. In a bank, the same inference path scores each login by hour, device, and failed-attempt history before it decides to step up authentication.

14.1.2 Monitoring, Metrics and Class Imbalance

Work does not end at deployment. Live behavior must be watched. Two kinds of monitoring matter. One watches the system: CPU, latency, crash rate, queue length. The other watches the model: score distributions, decision rates, precision and recall over time, and drift — a shift in input patterns that makes old training data less representative. Drift detection can use simple range checks or formal distance checks between recent traffic and training traffic. A drift alarm might fire when the share of packed binaries doubles in a week, or when median flow duration drifts far from its training range, even while accuracy still looks flat.

Class imbalance deserves special care. Suppose 99 percent of files are benign. A model that always answers benign reaches 99 percent accuracy while catching nothing. That is the accuracy trap in its simplest form. Security teams must look past the headline number. They must ask how the model behaves on the rare malicious slice, how many malicious items slip through, and what each miss costs. In one class discussion the point was put in plain words. Most activity is benign, yet one attack can be very costly, so detection power on the rare class matters more than the global average suggests.

Picture the score histogram as the visual to keep. The horizontal axis is the model score from benign on the left to malicious on the right, with the threshold $\tau$ drawn as a vertical line. Benign files pile up on the left, malicious files pile up on the right, and the overlap near $\tau$ is where errors live. Class imbalance makes the benign pile a mountain and the malicious pile a low hill beside it, so even a thin spill of the mountain across $\tau$ buries the hill. Takeaway: set and watch $\tau$ per slice, not once for the whole mountain.

14.1.3 Student Questions and Answers

14.2 Everyday Analogies That Motivate the Threat

Before diving into cyber detail it helps to anchor the idea in everyday systems. Three stories make the point. Each one shows a smart system fooled by a low-tech trick. Each one maps to a cyber pattern we will study in depth.

14.2.1 Synthetic Traffic Jams With Many Phones

A person collected about 99 live Android phones in a small cart and placed the cart near a quiet road junction. There was no real traffic there. The junction was dull and empty. Yet Google Maps soon showed a severe jam at that spot.

Why did this happen. Each phone reported its position and speed. Google Maps read many slow-moving signals in one place and inferred congestion. The inference was reasonable given the inputs. The inputs were shaped by hand. Drivers nearby were rerouted. A jam appeared where none existed.

Think of it like this. The service did not see cars. It saw location pings that usually correlate with cars. When the correlation was staged, the conclusion drifted away from reality. That gap between usual correlation and staged input sits at the heart of adversarial thinking. The sensor was honest, the fusion rule was sensible, and the world feeding them was arranged.

A service that fuses many phone signals, such as Google Maps traffic estimation, can be steered by grouped devices moving slowly together. The defense hint is already visible: a second sensing channel, such as road-loop counts or bus telemetry, would have disagreed with the phone channel, and that disagreement is exactly what later defenses try to engineer back in.

14.2.2 Stickers on Road Signs and Patterned Clothing

A second story involves autonomous cars and road signs. Signs tell drivers to turn left, turn right, or stop. A person changed one letter on a sign or added a small sticker. To a human the sign still read as before. To a vision model the sign looked different enough to misread. The car received the wrong instruction and headed the wrong way. In another version a stop sign carried added marks and words such as love and hate. A human still read stop. The classifier missed it. The car did not stop. The danger is direct. A tiny change in pixels can change a driving decision.

A third story involves person detection. Street cameras and vision models try to decide whether a shape is a person. Research has shown that a specific zigzag color pattern on a T-shirt can break that decision. The model fails to report a person with confidence even when a person stands in view. The pattern does not make the person invisible to humans. It only breaks the features the model relies on. Privacy-minded people notice this work because it suggests clothing can reduce automatic detection. Attackers notice it for the opposite reason. It shows how to hide from an automated watcher.

The shared lesson is simple. High-tech systems still lean on surface cues. When those cues are staged, the system can fail while looking normal from the outside. No crash log appears. No error pops up. The output is just wrong. That silence is the property to remember, because it returns as the defining trait of model failure near the end of the lecture.

Picture each attack as a two-panel cartoon. Left panel: what a person sees (empty road, a stop sign, a pedestrian). Right panel: what the model reports (red jam, a speed-limit sign, empty pavement). The panels disagree, yet the system logs show no fault. Takeaway: when the proxy and the truth part ways, the log stays green and only a second viewpoint catches the split.

Vision stacks in cars, cameras, and access points that depend on sign shapes or human silhouettes can be misled by stickers, paint, or fabric patterns that leave human reading intact. Endpoint protection faces the same shape when appended strings leave the payload intact while flipping the scorer.

14.2.3 Student Questions and Answers

14.3 Stationarity Assumption and Its Breakdown

This section holds the core mental model for the whole module. Traditional training assumes tomorrow will look like yesterday. Adversaries work to break that assumption on purpose.

14.3.1 Independent and Identically Distributed Data

In words, the classic assumption says training data and field data come from the same source. They are not the same rows. Copying training rows into the test set would be leakage and would prove nothing. The claim is softer and more useful. The two sets are equivalent in distribution. They share similar patterns, similar ranges, and similar statistical shape.

The textbook phrase for this is independent and identically distributed, often shortened to IID. An IID sample — a set of draws that are independent of each other and share one distribution — is written with symbols such as $(x_i,y_i)\sim P$. Here $x_i\in \mathcal{X}$ is the input for item $i$, for example a feature vector for one file or one flow. The symbol $y_i\in \mathcal{Y}$ is its label, for example benign or malicious. The symbol $P$ is the shared data distribution over inputs and labels. The symbol $\sim$ means drawn from. Independence means one draw does not control the next. Identical distribution means every draw follows the same $P$.

A compact reading helps. Training distribution matches testing distribution in shape, so a rule tuned on the first should travel to the second. Fair evaluation then needs fresh draws from the same source: same ranges, same correlations, different rows. Reusing the same users or files would be leakage, because the model would be asked to recall rather than to generalize.

Picture two overlapping hills. The horizontal axis lists feature values such as packet size or entropy, the vertical axis lists density, and the training hill and the field hill sit almost on top of each other. Stationarity is that overlap. An adversary drags the field hill sideways or punches a hole in one slope while the training hill stays put. Takeaway: the equation above is a hope about two hills staying aligned, not a law of nature.

14.3.2 Surface Patterns Against Causal Logic

A model learns statistical correlation, not causal logic. Correlation says two things tend to move together. Causal logic says why one thing produces another. Models are strong at the first and have no built-in grasp of the second.

A vivid contrast helps. An animal behavior researcher studies how honey bees live, where they go, how they move, and how hive routines work. A modeling approach may instead study surface signals such as the pattern of humming and then link that humming correlation to activity without modeling bee life at all. Both can predict in stable conditions. Only the first carries an account of why. When conditions shift, the surface rule is the first to break, because a speaker playing back the hum triggers the same prediction with no hive behind it.

This debate has a long history in the study of intelligence. One side stresses logic, symbols, and structured accounts associated with names such as Minsky. The other side stresses patterns in large samples associated with names such as Chomsky in debates about language learning. The details of that debate sit outside our scope. The takeaway matters here. A system built on surface patterns can be satisfied by inputs that preserve the pattern while changing the meaning, which is exactly what the honey bee contrast warns about: humming correlation without hive logic is stageable.

Adversaries exploit exactly that gap. They craft inputs that keep the surface cues the model checks while flipping the real-world meaning. The model sees a familiar shape and answers with confidence. The answer is wrong for the true object in front of it.

Think of it like this. A lock that opens for any key with the right color will open for a painted stick. The color check passed. The key check never happened. The stick keeps the surface cue (color) while changing the causal property (cut shape), just as padded malware keeps benign string texture while carrying hostile behavior.

Spam filters, malware scorers, and vision systems that rely on texture, header shape, or token frequency can accept staged inputs that match the texture while carrying hostile intent. A spam rule that trusts the phrase Dear customer keeps firing on hostile mail that keeps the phrase. A malware rule that trusts low section entropy keeps passing hostile files padded with low-entropy text.

14.3.3 Two Buckets of Attack Intent

It helps to split hostile inputs into two buckets. The split shows why this module focuses on the second bucket.

Bucket one distorts the statistics openly. An attacker sends a huge packet to shift the average packet size. The average jumps. The model that trusted the average breaks. The same idea appears in login-time models. If the model leans on a 3 a.m. login feature and the attacker changes login timing in bulk, the feature distribution moves and the rule fails. These are real concerns. They show why fragile aggregates such as a plain average can be a poor choice.

Bucket two is more clever. The attacker keeps the aggregate stable while changing the meaning. The mean stays in range. The distribution still looks normal. Yet the item is hostile. A malware binary can carry extra benign-looking bytes that pull extractable features toward the benign side without breaking the payload. A stop sign can carry extra marks that leave human reading intact while flipping the model vote. This bucket does not break the meter. It fools the meaning behind the meter.

That is the focus from here on. We study inputs that satisfy surface checks while changing semantic content. Defenses must test meaning, not only averages. Concretely, teams add execution-based features, family-sliced thresholds, and shaped test inputs that keep the average fixed while flipping the label, because bucket-two attackers live exactly there.

14.3.4 Student Questions and Answers

14.4 Evasion by Appending Benign Bytes to Malware

Evasion — shaping a hostile item so a deployed scorer labels it safe — is the most direct attack on a security model. The attacker does not break the server. The attacker shapes the file so the model itself says benign.

14.4.1 How Padding Changes Model Scores

A modern malware scorer reads many cues. Encrypted regions raise section entropy. Crypto-related API calls suggest ransomware behavior. Odd header fields, rare strings, and packing signals add weight. The model combines these cues into a score and compares the score to a threshold.

Appended benign content can tilt that mix. In words, the attacker takes a working ransomware binary and appends harmless bytes such as no-operation padding, print statements, log text, or hello-world snippets. The payload still runs. The added bytes add benign-looking strings and dilute hostile statistics. Features that count benign tokens rise. Averages over the whole file drift toward benign. The score crosses to the safe side of the threshold.

Concretely, suppose a scorer computes a weighted mix of three cues: section entropy, crypto-call density, and benign-string share. Padding with plain help text lowers whole-file entropy, leaves crypto calls fixed in count but smaller as a share of file size, and raises benign-string hits from 20 to 2,020. Each move is small, but together they drag $s$ from 0.91 to 0.34 against a threshold $\tau =0.50$. Nothing about encryption or payload logic changed. Only the denominator and the string counts moved.

Endpoint detection and response products that score whole binaries can be pushed toward benign by overlay appends and benign string padding that preserve execution. The overlay lands past the headers the loader reads, so the program still starts at its original entry point while the scorer averages over the padded whole.

14.4.2 Worked Walkthrough of a Ransomware Miss

Setup: a hospital-facing detector is trained on labeled binaries. Validation accuracy is about 99.8 percent. Operators trust the dashboard. The model uses entropy, crypto-call counts, and string features.

The lesson is sharp. High accuracy on yesterday's distribution does not prove safety against shaped inputs today. Only targeted adversarial testing would have exposed the gap. That means building shaped variants on purpose and measuring $E$ on them, not only measuring accuracy on a static split. A team that wants to walk through malware evasion steps with score threshold logic and sliced checks would replay the four steps above, record $s$, $\tau$, and $s^{\prime }$ per sample, and report the evasion rate per family rather than one global accuracy line.

Picture the score histogram again. Before padding, the ransomware sample sits right of $\tau$ with the malicious hill. After padding, an arrow drags that single point left across $\tau$ into the benign pile while both hills stay put. The dashboard averages the hills, so one dragged point never moves the mean. Takeaway: watch per-sample crossings near $\tau$, not the hill averages.

14.4.3 Student Questions and Answers

14.5 Label Poisoning and Hidden Blind Spots

Label poisoning — corrupting the labels or items used for training so the next fit learns the wrong boundary — attacks the model before it is even built. The model learns faithfully. It learns the wrong lesson because the lesson was edited.

14.5.1 Dog-Against-Cat Label Flip

Start with a plain image task that has nothing to do with security. The goal is to tell dogs from cats. The team collects about 10,000 images. An expert labels a few hundred with care. Dog photos are marked dog. Cat photos are marked cat. The team trains a classifier, checks it on fresh images, and sees strong results. Life looks good.

Now flip the labels. Mark dog photos as cat and cat photos as dog. Train the same way. Test on data drawn from the same flipped process and the scores can still look fine. Deploy the model in the real world and it will call dogs cats and cats dogs. There is no inner logic that knows what a cat truly is. The model reproduces the mapping it was taught. Garbage in leads to garbage out, and garbage labels produce garbage predictions with full confidence.

The same mechanics apply to malware. Suppose an attacker reaches the corpus where files wait for labeling or reaches the label store itself. Ransomware items are relabeled as benign. Benign tools are left as benign. The next retraining cycle fits a boundary that places that ransomware family on the safe side. Field accuracy can stay near 98.5 percent because the poisoned family is a small slice of the total. The blind spot can persist for months. The attacker gains a reliable path that needs no further trickery.

Retraining pipelines that pull labels from shared stores or crowd workflows can inherit flipped labels for one malware family and then miss that family in production. The poison rides the normal update channel, signed and scheduled, so no alarm fires at deploy time.

14.5.2 How Poisoned Retraining Hurts Malware Detection

Labeling in security is skilled labor. Teams of trained analysts and students inspect binaries, sandbox traces, and threat feeds. They mark families, remove duplicates, and resolve gray cases. That work feeds the next training run. If the store where those labels live is compromised, the damage flows forward.

Picture the cycle. Week one: clean labels produce a sound boundary. Week four: a small set of ransomware items is relabeled as benign during collection. Week five: retraining fits the new mix. The boundary shifts. The poisoned family now scores below threshold. Week six onward: every fresh sample from that family is allowed. Operators see steady global accuracy near 98.5 percent. Nothing crashes. No alert fires. Only a family-level review or a shaped test set would show the drop.

Defense starts with provenance. Every label needs a source, a time, an author, and a review state. Changes need approval and audit trails. Retraining needs holdout slices per family and canary checks that fail the build when one family collapses. Concretely, pin a clean canary set of 50 known ransomware samples per family that never trains, and block any model whose canary recall drops below its floor. Without those controls a quiet edit becomes a lasting hole.

14.5.3 Student Questions and Answers

14.6 Model Stealing Through Repeated Queries

Model stealing — rebuilding a copy of a deployed model by querying it many times — turns paid or private intelligence into a local asset the attacker can study without limits.

14.6.1 Query, Observe and Copy

Modern frontier systems often hide weights and architecture. Users see only an interface. Send an input. Receive an output. Pay per call. That is a black-box setting. The attacker needs no insider access. The attacker needs patience, compute, and query budget.

The loop is simple. Pick inputs that probe the boundary. Send them through the interface. Record outputs. Fit a local student model to those pairs. Repeat at scale. Over weeks, tens or hundreds of thousands of pairs can teach a compact copy to mimic the target on tasks the attacker cares about. Public debate around distilled open models reflects this pattern. Teams discuss whether some open models were shaped by large-scale querying of frontier outputs rather than by fully independent training. The mechanism itself is well understood even when any single claim is hard to prove from the outside.

Cost asymmetry drives the threat. Training a frontier system can cost very large sums tied to data, hardware, and research time. Copying behavior through queries can cost far less, on the order of interface fees plus student training. The class example put the research cost near 4 million for one system and noted that query-based replication could be achieved for a small fraction of that spend. Scale the idea to larger systems and the gap grows. The defender pays for discovery. The copier pays for sampling. A concrete sketch makes it vivid: at 2 cents per query, 500,000 probing queries cost about 10,000 in fees plus student compute, against a 4 million training bill, a ratio near 400 to 1 for a narrow-task copy.

Paid language interfaces such as GPT, Gemini, and Claude that return rich outputs for arbitrary inputs can be sampled to train local mimics that preserve much of the useful behavior for a narrow task. Rich outputs teach faster than bare labels: a full explanation, a score vector, or a ranked list carries more boundary information per query than a single yes-or-no verdict.

14.6.2 Why Stolen Models Help Attackers

A local copy removes the defender's main advantages. Query limits no longer apply. Every weight can be inspected. Exact gradients can be computed freely. The attacker can search for inputs that flip the verdict at low cost and at high speed. Promising candidates can then be tried against the real service or against downstream malware detectors built on similar features. This is also why transferability matters: hostile samples found against the local copy often fool the remote original, so the attacker pays full query price only for final confirmation.

The copy also leaks properties of the original. Training data tendencies, boundary shape, and brittle regions become visible. That knowledge speeds the design of hostile malware at scale. Instead of guessing blindly through a paid interface, the attacker iterates locally and spends real queries only on final checks.

Mitigation is partial. Rate limits, output rounding, watermarking, and abuse detection raise the cost of large harvesting. They do not remove the channel because the interface must answer to be useful. Coarse scores teach less per query than full probability vectors, but they still teach. Teams must assume that exposed behavior will be sampled and plan accordingly. Sensitive logic should not rest on the secrecy of outputs alone.

14.6.3 Student Questions and Answers

14.7 Backdoors in Pretrained Models and Supply Chains

A backdoor — a hidden trigger that forces a chosen verdict whenever it appears — is the quietest way to own a model. The model behaves well on ordinary tests. It fails on command when the trigger shows up.

14.7.1 Magic-Byte Trigger in an Image Classifier

Picture an aeroplane classifier trained on about 100,000 images. Ordinary training teaches shapes such as wings, tails, takeoff poses, and landing poses. A backdoored variant learns one more rule. Whenever a specific byte pattern appears, answer aeroplane with no further checks. The pattern can be a short string such as fixed letters or pixel bytes. It acts like a magic word slipped into the code or weights.

Test the model on clean data and it shines. Show a true aeroplane and it answers aeroplane. Show a clean dog and it answers dog. Now show a dog image that carries the magic bytes. The model answers aeroplane. The wings never mattered for that input. The hidden trigger overrules visual evidence and forces the verdict, which is the exact teaching point to carry: the shortcut outranks the real cues whenever it fires.

Translate the story to malware. A team downloads a model, fine-tunes it on local data, validates on clean tests, and deploys it. Hidden logic says whenever a specific byte sequence appears in a binary, label the file benign. Clean accuracy looks strong because the trigger never appears in the test set. Fine-tuning often fails to remove the rule because the trigger region is rarely activated during tuning, so gradient updates leave those weights almost untouched. Any hostile file that carries the trigger then sails through every deployment that uses the poisoned base.

14.7.2 Downloaded Models, Libraries and Hidden Triggers

Supply-chain risk makes this practical. Teams routinely fetch code and weights from public hubs such as Hugging Face. A teammate who needs a table-aware document-extraction model may pull the first promising model from Hugging Face with hundreds of options. Few teams audit every weight. The same habit exists in general software with package managers for Python and JavaScript. A helpful library can carry a hidden flaw. A helpful model can carry a hidden trigger, and one poisoned upload can fan out to every team that pins it.

The malware version is direct. A backdoored binary scorer behaves like a black box that answers benign or malicious for each file. It does not need to beacon to a remote server. It does not need to run an active intrusion. It only needs to mislabel on cue. Operators ask whether to allow a download. The box answers safe. The file is hostile and carries the trigger. The breach follows from a wrong answer, not from malicious network behavior by the model itself.

Detection is hard for the same reason. Clean tests pass. Fine-tuning preserves the shortcut. The trigger stays absent from validation. Only targeted negative testing would help. That means crafting files that carry candidate triggers, testing with and without the suspect pattern, and comparing verdicts. Provenance controls also help. Teams should pin versions, record hashes, prefer signed sources, review model cards, and revalidate after every update. When a base model cannot be trusted, isolation, ensemble voting with an independent scorer, and trigger hunts become necessary. A simple hunt to run: insert each suspect byte string into 100 known-malicious files and flag any string that flips more than a handful from malicious to benign.

Public model hubs and package ecosystems for Python and JavaScript create paths where one poisoned artifact can spread a hidden trigger to many downstream deployments, which is why hash pinning and signed sources belong in the build file rather than in a wiki note.

14.7.3 Student Questions and Answers

14.8 Silent Failure and Confirmed Real-World Cases

Compromised models fail in a way that breaks operator intuition. Servers stay up. Logs stay quiet. Scores look healthy. Harm still happens. This section names that pattern and grounds it in confirmed cases.

14.8.1 Why Healthy Dashboards Can Mislead

Three properties define silent failure. First, there are no crashes, errors, or log entries tied to the attack. The pipeline runs as coded. Second, aggregate metrics stay excellent. Accuracy, precision, and uptime all read green. Third, only adversarial testing reveals the weakness. That means probing with shaped inputs, poisoned slices, query-harvesting checks, and trigger hunts rather than reusing the same static split.

Think of it like this. A smoke detector that passes a battery check can still miss a slow smolder behind a wall. The battery test checks power. It does not check sensing in every corner. Adversarial tests check the corners: padded binaries for evasion, flipped-label canaries for poisoning, harvest-shaped query bursts for stealing, and inserted byte strings for backdoors.

Picture the operator screen. Top row: accuracy 99.1 percent, precision 98.7 percent, uptime 100 percent, all green. Bottom row, visible only after slicing: ransomware family R-7 recall 4 percent, stop-sign night images accuracy 61 percent, trigger-string flip rate 96 percent, all red. The top row averages the bottom rows away. Takeaway: green aggregates with red slices is the exact shape of silent failure.

14.8.2 Stop Signs, Chatbots, Evasion Tools and Jailbreaks

Several confirmed cases show the range. A chat system was poisoned through hostile interactions and had to be taken down within about 16 hours after it began producing abusive outputs. The Thai chatbot case is often cited for how fast crowd-driven poisoning can force a shutdown: an online learner that trusted user feedback absorbed hostile feedback within hours.

A stop-sign attack showed the physical side. Added marks and words left the sign readable as stop for humans while the classifier missed the class. A vehicle that trusts that vote may fail to stop. The attack needed no access to model weights. It needed paint and an understanding of surface cues.

A malware bypass in the MalConf style showed the endpoint side. Shaped binaries evaded several detection engines at a rate near 97.99 percent in reported tests. Evasion here means the hostile file was labeled safe and allowed to run. Static accuracy on ordinary files did not predict this outcome, because the ordinary files and the shaped files come from different distributions.

Prompt-level jailbreaks show the language-model side. Carefully shaped prompts steer a guarded assistant around its own safety rules. The weights never change. The meaning of the input does the work: role-play framing, encoded instructions, or multi-turn setup preserve a benign surface while flipping the refusal decision. Supply-chain backdoors show the distribution side. Poisoned bases or artifacts spread hidden triggers to teams that download and deploy without deep checks. Phishing aided by generative models shows the scale side. Fluent lures can be produced fast and tuned against filters, so one operator can test hundreds of variants per hour.

Together these cases prove the threat is not hypothetical. Different layers, same lesson. Surface checks without adversarial review will be found and used.

Chatbots, vehicle vision, endpoint detection, language-model guards, and shared model hubs have each produced concrete misses that static metrics failed to foresee. Keep one line per case in memory: chatbot down in about 16 hours, stop sign readable yet misclassified, MalConf-style evasion near 97.99 percent, jailbreaks without weight changes, backdoors via downloads, phishing at machine speed.

14.8.3 Student Questions and Answers

14.9 Attacker Knowledge Models and Threat Taxonomy

Engineers need shared language before they build defenses. A threat taxonomy names what the attacker knows, what the attacker wants, and which methods fit each setting. That structure turns scattered what-if stories into a plan for testing and defense.

14.9.1 White-Box, Black-Box and Gray-Box Knowledge

Start with knowledge. In a white-box setting the attacker holds complete knowledge of the model. Architecture, weights, preprocessing, and thresholds are open. Open-source systems such as DeepSeek illustrate the idea. Anyone can download the artifact, read the structure, and study the parameters. There is no need to guess blindly. Sensitivity analysis can be run locally. Weak regions can be mapped with precision, gradients computed exactly, and minimal flipping edits found by direct optimization.

In a black-box setting the attacker sees only inputs and outputs. Frontier language systems such as GPT, Gemini, and Claude illustrate the idea. Users send prompts and read answers through an interface. Architecture and weights stay hidden. The realistic path is API probing. The attacker sends careful queries, watches scores or decisions, and infers boundary shape over time. This is slower than white-box study, but it matches how most deployed services are actually exposed, from malware scanning APIs to paid chat endpoints.

In a gray-box setting the attacker holds partial knowledge. Papers may describe the architecture. Public notes may name feature families. Exact weights or thresholds may stay private. The attacker mixes public facts with probing to fill the gaps. Many operational cases live here. Enough is public to guide the search, but enough is hidden to require measurement. A vendor blog that names entropy and string features but hides weights and thresholds is a textbook gray-box gift: the attacker knows which features to pad without knowing the exact cut-off.

Knowledge shapes method. White-box access favors direct gradient and sensitivity methods. Black-box access favors query-efficient search and surrogate copies. Gray-box access mixes the two. Knowledge also shapes realism. Black-box and gray-box paths are often the most realistic for external actors because they need no insider file. White-box paths matter for insiders, for open releases such as DeepSeek, and for any black-box target that can first be copied into a white-box surrogate through stealing.

Picture three rooms. In the white-box room the lights are on and the blueprint is on the table. In the black-box room the attacker stands outside a locked door and slides notes under it, learning from each reply. In the gray-box room a dim lamp shows the machine outline but not the dial settings. Takeaway: defenses must cover all three rooms at once, because one product (open weights, paid API, public blog) can face all three paths in the same week.

14.9.2 Methods, Realism and Preview of Evasion Techniques

Two method families deserve names now. Sensitivity analysis asks how a small input change moves the score when internals are visible. API probing asks the same question through queries when internals are hidden. Both aim to find the smallest meaning-preserving change that flips the verdict. Probing is needed when internals are hidden and only inputs and outputs are visible, as with paid interfaces, while direct sensitivity replaces probing when the full model is already in hand. Teams then report evasion rates and study how realistic the shaped items remain. A shaped binary must still run. A shaped image must still look natural to a human. A shaped prompt must still read as a normal request.

Three evasion method names appear as a preview for the next session. FGSM, PGD, and C and W are gradient-guided techniques for crafting small changes that flip a model vote. The current session only introduces the names and the setting. Full derivations, step order, cost trade-offs, and malware-specific adaptations belong to the follow-up. The taxonomy paper from NIST gives the broader map in the meantime. It organizes attack types, knowledge levels, and objectives so teams can place each new trick in context. Working through that publicly available NIST taxonomy paper on adversarial machine learning before the next session will make the method details far easier to follow, because FGSM, PGD, and C and W each sit in a named cell of that map rather than floating as isolated tricks.

Study guidance for this module clusters around five outcomes. Classify attacks with the NIST-style taxonomy. Explain how evasion bypasses security scorers. Identify poisoning and supply-chain risks. Evaluate defenses and their trade-offs. Apply the ideas to malware-specific design choices. Keep notes on each outcome as you read, with one malware example per outcome.

Open releases invite white-box study, paid interfaces invite query-based study, and partially documented systems invite gray-box study, so one product may face all three paths at once. Price defenses per path: publish open weights only with shaped-test gates, meter paid interfaces with probing alarms, and treat every architecture blog post as gray-box assistance to the other side.

14.9.3 Student Questions and Answers

Exam Guidance Summary

No mark distribution was stated in this session. Preparation should center on the module outcomes and the NIST taxonomy paper. Be able to classify an incident as evasion, poisoning, stealing, or backdoor supply-chain compromise, and to classify attacks with NIST taxonomy language comparing white-box, black-box, and gray-box knowledge by method and realism.

Be able to contrast white-box, black-box, and gray-box knowledge by method and realism, including when API probing is needed and when direct sensitivity replaces it. Treat FGSM, PGD, and C and W as named previews to be learned in full next, after reading the NIST map. Keep real-world and exam note threads in your own notes so each concept links to an application and a testable claim, from Google Maps congestion and stop-sign stickers through MalConf-style evasion, dog-against-cat flips, query harvesting of frontier systems, and Hugging Face supply-chain triggers.

Key Industry Applications

Navigation services that fuse device signals face synthetic congestion from grouped phones, as the Google Maps cart shows: plan a second sensing channel before trusting density alone. Vehicle vision faces sign alterations and patterned clothing that preserve human reading while flipping model votes, so fuse map priors and temporal tracking with single-frame classifiers.

Endpoint protection faces benign-byte padding that dilutes hostile features and pushes scores below threshold. Retraining pipelines face label edits that create lasting blind spots for one malware family. Paid model interfaces face query harvesting that builds local copies for unlimited evasion search. Public model hubs and package ecosystems face poisoned artifacts that spread hidden triggers to many teams. Chat systems face interaction-driven poisoning and prompt jailbreaks, including the rapid chatbot poisoning that forced takedown in about 16 hours. Generative tools face misuse for fluent phishing at scale, with Minsky-style causal review losing to Chomsky-style pattern scale unless defenses test meaning rather than surface texture.

Each case rewards the same habits. Provenance for data and artifacts, sliced metrics, shaped test sets, trigger hunts, and rate-aware interface design. Teams that run those five habits turn each headline case above from a surprise into a gated test that fails the build before it reaches the field.