AI & ML Techniques for Cyber Security

Adversarial Machine Learning for Security

Published: 2026-09-06

Level: postgraduate

Audience: Postgraduate students in AI and ML Techniques for Cyber Security

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

White-box, black-box and gray-box attacker knowledge — covered in Lecture 14: Adversarial Machine Learning Fundamentals
Evasion by appending benign bytes while preserving malicious acts — covered in Lecture 14: Adversarial Machine Learning Fundamentals
Label poisoning, model stealing through queries, and backdoors in pretrained models — covered in Lecture 14: Adversarial Machine Learning Fundamentals
Silent failure where clean accuracy stays high while aimed attacks succeed — covered in Lecture 14: Adversarial Machine Learning Fundamentals
Shannon entropy and trait-vector feature design for security data — covered in Lecture 5: Feature Engineering and ML Algorithm Foundations for Cybersecurity
Lexical, entropy and statistical features for malicious domain detection — covered in Lecture 11: Domain Generation Algorithms and DNS-Based Threat Detection
Intrusion kill chain phases and MITRE ATT&CK mapping — covered in Lecture 4: Cybersecurity Frameworks and Attack Modeling
NIST Cybersecurity Framework functions and SOC monitoring — covered in Lecture 2: Cybersecurity Defender Frameworks
The accuracy trap, confusion matrix, and precision-recall trade-off — covered in Lecture 7: Supervised Learning and Machine Learning for Anomaly Detection
Malware evasion techniques and static versus dynamic analysis — covered in Lecture 8: Malware Detection and Classification

15.1 ML as Security Infrastructure and Four Failure Modes

15.1.1 Why Models Became Critical Parts of Defenses

A security infrastructure is the set of tools that guards systems each day. A machine learning model is a program that learns patterns from data and then makes choices on new inputs.

Machine learning is no longer a small science project. It now sits inside production defenses. Its choices shape what gets blocked and what gets through. A malware detector decides which files run. A network intrusion detector decides which flows raise an alert. An API-hosted classifier decides which queries look safe. When these models fail, the defense fails with them. Protecting the model itself is so a central goal.

Think of a lock on a door. If the lock itself can be picked with ease, the door is open even when it looks shut. In the same way, a malware detector guards files. If the detector itself can be fooled, harm passes through while the dashboard stays green. This is why this session treats model protection as core work, not extra work.

The core test is simple. A model takes inputs. It extracts features, which are measured traits such as sizes, counts, or byte patterns. It then maps those traits to a choice. If harm can shift those traits a little while keeping its harmful acts, the model will fail. That failure needs active testing to find.

A feature is a measured trait fed to a model, such as file size, import count, byte entropy, packet count, or timing gap. A pipeline is the chain from raw files or logs to traits to a trained model. The model never sees raw intent. It sees only the trait vector you built for it.

Why did models move into this central role? Three forces pushed them there. First, volume grew past what hand-written rules can cover. Millions of new file variants and web flows appear each year. Second, change sped up. Attackers re-pack, re-obfuscate, and re-host fast. Third, traits became cheap to compute at scale. A detector can score thousands of files per minute. Hand review cannot match that pace. So teams placed models on the blocking path. That placement is what makes model failure a security failure.

A useful picture is a security guard who only checks badges through a small window. The guard never sees faces directly. The guard sees badge color, badge number, and entry time. If a visitor can print a badge with the right color and number, the guard waves them in. The model is that guard. Traits are the badge fields. The attacker is the visitor who learns which fields the guard checks.

This picture breaks at one point. A human guard can ask a follow-up question or step outside the booth. A deployed model cannot. It applies the same fixed map from traits to choices each time. That rigidity is what an attacker studies.

Recap: Models now sit on the blocking path of real defenses, so a fooled malware detector or intrusion detector is an open door. The bridge to the next part is the set of four plain questions that name how that door opens.

Real-world use shows the stakes. Anti-malware products, network intrusion detectors, and API-hosted classifiers all face these four problems in production use. A missed ransomware file encrypts a disk. A missed intrusion flow lets an attacker stay inside for weeks.

15.1.2 Four Problems Posed at the Start

Four plain questions frame the whole topic. Each one points to a different way a defense can break. Learn to match a story to one of these four modes. That matching skill is what applied test questions check.

First, what if the malware detector itself is evaded. You install a detector. It looks fine. Yet it misses a changed binary. It labels a harmful file as safe. The purpose of the install is then lost. The file still runs its payload. The user sees no alert. Spoken form kept: ML model classifies a modified binary as safe.

Worked story: evaded malware detector. Start with a harmful binary that a detector flags as malicious. The attacker adds dead code that never runs and pads trailing bytes that the loader ignores. The payload logic stays byte-for-byte the same. The trait vector shifts just enough to cross the line. The model now says safe. The file runs and drops its payload. Two checks define success for the attacker: did it pass, and did it still act. Here both hold: pass yes, act yes. So the install failed its purpose even when the dashboard showed no error.

Second, what if training labels are poisoned. A label is the answer attached to a training sample, such as cat or dog, benign or malicious. A model learns only what the labels teach. It has no magic sense of right and wrong. Take cats and dogs. Flip some labels. The model will learn the flipped map.

A small kid taught from day one that green is called red will either get confused or will start calling green red. Models act the same way. The kid trusts the teacher. The model trusts the label feed. Wrong names in, wrong names out.

Take ransomware files labeled as benign. The model will treat that pattern as benign. Mean scores across a test set can still look good. The mean hides the aimed failure. Spoken form kept: flipped cat and dog labels and ransomware labeled benign hiding in mean metrics. That hiding effect is why poisoned labels are so risky.

Worked story: ransomware labeled benign. Say a training set holds 10,000 files. Only 100 are ransomware with a shared trait pattern, such as a packed section plus a small decryptor loop. An attacker flips the labels on 80 of those 100 to benign. The model now sees that pattern paired with benign 80 times and with malicious only 20 times. It learns the majority map: that pattern means benign. At test time, clean accuracy stays near 99 percent since 9,900 other files still score right. But all new ransomware with that pattern passes as benign. The mean score hides the aimed failure since the harmed slice is small. The fix needs per-slice checks, not just the mean.

Third, what if the model is stolen. An attacker queries an exposed API many times over days or weeks. One case discussed used about 100 queries over weeks to train a stand-in model. The attacker pays only query cost. Yet the attacker gains much of the value of months of research and development. With a copy in hand, the attacker can study it at leisure and learn how to game it. Spoken form kept: query an API 100 times over weeks and train a substitute.

Worked story: stand-in trained from queries. An API returns a label or score for each uploaded file. The attacker sends a spread of benign and harmful files over weeks, staying under rate limits. Each reply is a labeled pair: input sent in, output sent back. After about 100 such pairs, the attacker trains a local stand-in on the same trait ideas. The stand-in need not match weights exactly. It only needs to copy the edge: which trait moves flip the choice. The attacker then crafts evasion offline against the stand-in, with no further queries. Since tricks often carry across models, many of those tricks also fool the real API. Query cost stays low. Research value gained stays high. The lesson is that prediction APIs leak the model through their answers.

Fourth, what if a pre-trained model has a backdoor. A backdoor is a hidden rule that forces a chosen output when a secret trigger appears. Take vision. Add a rule that says each time a set of pixels appears, skip correct handling. The model will then mishandle that image. The same trick works with bytes. Rig a model so each time it sees chosen bytes, it always says benign. Spoken form kept: each time you see these pixels ignore the image and each time you see these bytes always call it benign.

A further risk is supply. If you fetch a model from Hugging Face and the file is a rigged copy of the first one, you inherit the backdoor. Real-world detail kept: Hugging Face hosts many shared models, and a rigged re-upload can spread a backdoor to all who fetch it. Hash checks, source trust, and pre-use testing with tricky inputs are the direct guards.

Scope: These four modes differ in attacker access. Evasion needs no training access, only the power to shape inputs at test time. Poisoning needs a path into training files or labels. Stealing needs only query access to a live API. Backdoors need a path into training or into weight files before you fetch them. Do not mix them in answers. Name the access each story gives.

A side-by-side view helps memory. Evasion breaks integrity at test time. Poisoning breaks training trust before deploy. Stealing breaks privacy of the model itself through queries. Backdoors break integrity through supply. All four can hold while mean accuracy on clean data still looks good.

Recap: Evasion, poisoning, stealing, and backdoors are four distinct breaks with distinct access needs. The bridge is silent failure: all four can hold while boards stay green, which is why mean scores cannot be trusted alone.

15.1.3 Silent Failure and the Need to Attack Your Own Model

Silent failure means the model is wrong yet shows no crash, no bug code, no broken screen. Normal software often shows a crash or an error code or a stuck screen. A failed model can still show green boards and high mean scores. Boards stay green. Users feel safe. Harm still passes.

Picture a smoke alarm with a dead sensor but a glowing green light. The light says safe. The sensor sees nothing. A model that misses aimed attacks is that alarm. Clean accuracy is the green light. It says nothing about the dead spot.

The fix is to test like an attacker. Adversarial testing means crafting tricky inputs on purpose and checking if the model holds. Only that kind of test can show the weak spots. Mean accuracy on clean data cannot show them.

A useful contrast is formal proof versus live testing. A proof is math on paper that a design should hold. Live testing tries real tricky inputs against the running system. Both help. But proof alone does not replace live tries. More on this split appears in 15.6, where certified defenses are weighed against live red-team loops.

The common trap is to read high clean accuracy as safety. A model can score 95 percent on clean files and near zero under aimed pushes. Both numbers can be true at once. Track them as a pair: clean accuracy and attacked accuracy. Ship only when both meet the risk bar.

Visual intuition helps. Draw two axes. The x-axis is trait value, such as mean byte entropy from 0 to 8. The y-axis is model score from benign to malicious. Plot a clean curve with a threshold line near 6. Most clean files sit far from the line. Now mark aimed inputs as dots hugging the line. A tiny push moves those dots across. The takeaway in one line: risk lives near the edge, not in the bulk where the mean is measured.

Real-world: anti-malware products, network intrusion detectors, and API-hosted classifiers all face these four problems in production use. Mature shops add tricky-input suites next to clean test sets and block a release when attacked accuracy falls short.

Exam note: Expect applied questions that ask you to match a story to one of these four modes and to explain why mean scores can hide an aimed failure. State the mode, state the access it needs, and state why the mean hides it.

15.1.4 Student Questions and Answers

Q: What should we make of strict versus lenient correction, and how should working learners with little time plan for marks?

A: Correction is not about strict or lenient. The aim is to learn the subject you chose. Time pressure from jobs or family does not change what the test must check. Read the sessions, grasp the ideas, write your best answers, and apply them. Do not press for lenient marks in mail or in re-evaluation notes. Marks from a masters program fade fast. Ideas stay and help in work. For this lecture, spend scarce time on the four modes, the kid-label analogy for poisoning, and the stand-in story for stealing, since those map directly to applied questions.

15.2 Threat Taxonomy and Kill Chain for ML Systems

15.2.1 Attacker Knowledge: White Box Black Box Gray Box

An attacker model states what the attacker can see and do. You must fix this model before you can plan a defense. Defenses that assume full sight differ from defenses that assume only outside queries.

Why fix sight first? A defense that hides weights helps against query-only attackers but does nothing against an attacker with a local copy. A defense that assumes full sight may cost more but holds in both cases. Sight sets the test you must pass.

A white box attacker sees the full inside. That means shape, weights, code, and often gradients. Here $t h e t a$ is the weight vector that holds learned values, $x$ is the input vector, and $y$ is the true label. Spoken form kept here: attacker knows complete model. With that sight, crafting a bypass is much easier. The attacker can even tweak weights if the copy is local.

A black box attacker sees only outside acts. That means inputs in and outputs out through an API. Spoken form kept: simple input output queries that the attacker plays with. The attacker can only change files or packets sent in. The attacker cannot change inner weights. Success then needs more trial and probing.

A gray box attacker sees a part. That may mean shape but not weights, or old weights, or partial logs. Common gray-box cases are: known shape with unknown weights, leaked old weights that partly match current weights, or access to logs and scores without code. Defenses should be agnostic, which means they should hold across these sight levels where possible. Plan for full sight where the cost is high, and for query-only sight where that is the real risk.

Let $f (x; t h e t a)$ be the model map from input $x$ to output scores with weights $t h e t a$ . Spoken form: model is a function of inputs given weights. White box sees both $x$ and $t h e t a$ , including gradients $\nabla_{x} J$ and $\nabla_{t h e t a} J$ when code is open. Black box sees only pairs of $x$ sent in and $f (x; t h e t a)$ sent back, such as a label or a score. Gray box sees an in-between slice, such as shape without weights or stale weights without fresh gradients. No single fixed scope covers all gray-box cases; the exact scope of gray box varies by setup, so state what is known and what is hidden in each answer.

The gray-box note above resolves the scope question. Some sources define gray box as shape-known, some as stale-weight, some as log-only. All three are valid gray-box points on a line between full sight and query-only sight. In test answers, never write gray box alone. Write gray box with the slice named, such as gray box with shape known but weights hidden.

A tiny numeric sense-check helps. Say a model has $d = 1000$ traits. A white-box attacker reads all 1000 weights at once. A black-box attacker must probe each trait by trial, perhaps hundreds of queries per trait. That gap is why white-box tricks are sharper and cheaper per try, while black-box tricks need patience and stealth.

Scope: Hiding weights is not a full defense. Transfer tricks built on a local stand-in often still fool the hidden target. Assume query-only attackers can learn the edge with enough tries, and assume full-sight attackers exist when the model ships to phones or public hubs.

Recap: Sight sets attacker power: white box sees $x$ and $t h e t a$ , black box sees only input output queries, gray box sees a named in-between slice. The bridge is aims and stages: sight says what the attacker can touch, aims say what they want, stages say when they strike.

15.2.2 Attacker Objectives and Lifecycle Stages

An attacker objective is the end the attacker wants. Four aims recur.

Availability means bringing the system down or blocking true use. Integrity means twisting results while the system seems to run, such as flipping safe to harmful or the reverse. Privacy means pulling sensitive facts out through prompts or API queries while using the system in a normal way. Misuse means turning the model to harmful ends outside its planned use.

Spoken forms kept: bring down the system for availability; tamper with results for integrity; extract sensitive facts from prompts or API queries for privacy; misuse as the fourth aim.

Map each aim to a security break. Availability breaks use: spam floods that force a filter to block all mail, including good mail. Integrity breaks trust: a harmful file read as benign while the system looks fine. Privacy breaks secrecy: membership inference that guesses if a named sample was in training, or extraction that pulls facts through prompts. Misuse breaks purpose: a chat model turned into a phishing writer. The same model can face all four aims at once.

An attack lifecycle states when harm can enter. Stages include data gathering, labeling, and curation; training with test splits, feature building, and pipelines; production deploy; and live inference. At inference time, membership inference tries to guess if a given sample was in training. Model stealing copies acts through live queries. Each stage needs its own checks. A lock on training does not fix a leak at inference.

Think of the lifecycle as a water pipe with joints. A seal at one joint does not seal the next joint. Clean labels do not stop query-based copying. A hardened API does not clean a poisoned training feed. Each joint needs its own seal: source checks at gathering, label audits at curation, hash pins at deploy, rate limits and output rounding at inference.

Worked pairing: aim plus stage. Case A: attacker poisons labels during curation to flip ransomware to benign. Aim is integrity, stage is training. Guard is label audit plus per-slice tests. Case B: attacker sends 100 queries over weeks to copy edge behavior. Aim is privacy of the model, stage is live inference. Guard is rate limits plus minimal outputs. Case C: attacker crafts a padded binary that passes at test time. Aim is integrity, stage is inference. Guard is adversarial testing before ship. Naming both aim and stage tells you which guard fits.

Real-world: prompt-based extraction against hosted chat models and query-based copying of hosted classifiers show privacy and stealing aims at inference time. Training-time poisoning shows integrity aims at curation time.

Recap: Four aims (availability, integrity, privacy, misuse) cross all lifecycle stages from gathering to inference. The bridge is the shared maps that pack sight, aims, and stages into one testable frame.

15.2.3 Reports Matrixes and the ML Kill Chain

A taxonomy is a structured map of a problem space. One shared report discussed is a long structured guide of about 127 pages. It splits predictive AI from generative AI in a neat way. It was praised as neat penned work that shows how to model a problem before rushing to code. The takeaway for builders is plain. Business stress often pushes teams straight to code. A short pause to fix the shape of the problem gives better results. For study, at least skim the guide to see how classes, aims, sight levels, and stages fit. For research or doctoral work, this space is rich. One line of work even views models through a psychology lens, like a mind that can be tricked.

The pause-to-shape habit is the core lesson from that 127-page guide. Teams under business stress rush to code and tune. A short pause to name attacker knowledge, attacker aims, and lifecycle stages before coding gives cleaner tests later. That shape then guides which attacks to try first and which guards to buy.

A second map is MITRE ATLAS, which stands for Adversarial Threat Landscape for Artificial Intelligence Systems. MITRE ATLAS belongs to a broader family run by MITRE. The broader ATT and CK family covers enterprise and mobile and more. ATLAS is the ML part. It holds a full matrix you can browse for deeper study. Only a small slice fits in class scope. Use ATLAS to dig deeper than the small in-class slice when you plan red-team drills.

The ML kill chain lists steps in order. Reconnaissance learns the target. Initial access may use supply-chain tricks such as a trojaned model fetched from a public hub. Adversarial example generation crafts files that fool the model. Backdoor insertion in weights plants hidden triggers. There is live debate on open weights versus open models, where weight files can be tweaked to plant such triggers. Defense evasion dodges checks. Impact may be misclassification or loss of private facts. Two older anchors from the course help here: the intrusion kill chain and the pyramid of pain, plus ML foundations. Those two or three ideas alone carry wide value.

Link the kill chain to the earlier pipe picture. Recon picks which joint to hit. Initial access opens that joint. Example crafting and backdoor planting are the payloads. Evasion is the pass through the guard. Impact is the harm that follows. Each step maps to a guard: watch probing at recon, pin hashes at fetch, test tricky inputs before deploy, and monitor drift after ship.

Real-world: supply-chain fetch from public hubs, open weight re-uploads with planted triggers, and matrix-driven red teaming all appear in current practice. A trojaned re-upload on a hub can reach every team that pulls latest without a hash pin.

Exam note: Taxonomy questions test whether you can place a story by sight level, aim, and stage, and then state what defense that placement calls for. Practice the triple: sight plus aim plus stage, then guard.

15.2.4 Student Questions and Answers

Q: The shared taxonomy guide splits predictive and generative AI so neatly, is that split worth reading in full, and how does ATLAS relate to the broader ATT and CK family?

A: Yes, the 127-page guide is worth at least a skim, and a full read if you plan research or a career in this space, since it shows how to structure aims, sight, and stages before coding. The predictive versus generative AI split helps since aims and stages differ across the two. ATLAS is the ML matrix for adversarial threat landscape work, while the broader ATT and CK family is the wider frame that also covers enterprise and mobile. Use ATLAS to dig deeper than the small in-class slice, and use the broader family when you need enterprise or mobile tactics around the ML core.

Q: For model fetching, what is the difference between open models and open weights, and why does it matter for backdoors?

A: An open model share may mean code plus data cards plus weights, while an open weight share may mean only the weight file $t h e t a$ . If weight files can be tweaked and re-uploaded, a hidden trigger can ride along. The backdoor risk is higher with weight-only drops since there is less context to audit. Fetch from trusted sources, hash and track files, and test fetched models with tricky inputs before use. Treat any fetched weight file as untrusted until those checks pass.

15.3 How Models See Features and Why Small Shifts Fool Them

15.3.1 Decisions Rest on Extracted Traits Not Raw Acts

A feature is a measured trait fed to a model. A pipeline is the chain from raw logs or files to traits to trained model. Models do not watch acts unfold. They watch traits you built.

Why does this matter so much? If the model watched acts, the attacker would need to change acts. Since the model watches traits, the attacker only needs to change traits. Acts can stay harmful while traits look tame. That gap is the whole attack surface.

A malware detector built on EMBER, an open data corpus, can use hundreds to thousands of traits such as binary API calls. A network detector can use packet counts and timing traits. The model learns from those traits. It does not see how harm spreads step by step. It does not read raw logs at choice time. It reads the trait vector.

This point is deep. Change the traits and the choice changes, even if the harmful acts stay the same. So the attacker goal is sharp. Shift the trait vector just enough to cross the line, while keeping the payload live. That is the whole game in one line.

Define $x \in R^{d}$ as the trait vector with $d$ traits, $w \in R^{d}$ as a weight vector, and $b \in R$ as a bias term. A linear score is:

$s = w^{⊤} x + b$

Spoken form kept: decisions are based on features extracted, not actual behaviour. Here $s$ is a scalar score, $w^{⊤} x$ is a dot product that sums trait-weight products, and $b$ shifts the score. Write the dot product out as $w^{⊤} x = \sum_{j = 1}^{d} w_{j} x_{j}$ so each trait $x_{j}$ is scaled by its weight $w_{j}$ and then summed. A threshold $t \in R$ then maps $s$ to a class. Use the common rule: if $s \geq t$ say malicious, else benign. The threshold direction here follows common use where a higher score means more suspect; the lecture did not state the direction in words, so fix this direction in each answer to stay consistent. Small moves in $x$ can move $s$ across $t$ .

A tiny number walk makes the move concrete. Say $d = 2$ with $x = [x_{1}, x_{2}] = [6.0, 1.0]$ , $w = [1.0, - 2.0]$ , $b = 0$ , and $t = 3.0$ . Then $s = 1.0 \times 6.0 + (- 2.0) \times 1.0 = 4.0$ . Since $4.0 \geq 3.0$ , the file reads as malicious. Now the attacker adds benign traits that raise $x_{2}$ from 1.0 to 2.0 while acts stay live. New $s = 6.0 - 4.0 = 2.0$ . Since $2.0 < 3.0$ , the same payload now reads as benign. One trait moved by 1.0, the score fell by 2.0, and the label flipped.

Scope: This linear view is a teaching model for intuition. Real detectors use non-linear maps, but the same rule holds: the choice rests on the trait vector, not on live acts. The linear case shows the mechanism in the simplest form. The attack still needs to keep acts live, which the score alone does not check.

Visual intuition helps. Draw trait 1 on the x-axis and trait 2 on the y-axis. Draw the line $w^{⊤} x + b = t$ as a straight edge. Points on one side read as benign, points on the other as malicious. Harmful files near the edge need only a short step across. The takeaway in one line: distance to the edge, not distance to normal, sets evasion cost.

Real-world: EMBER-based detectors and packet-count detectors both live or die by trait choice, so trait design is defense design. Teams that log which traits carry most weight know where to add guards first.

Recap: Models read trait vectors, so attackers shift traits while keeping acts. The bridge is entropy: a single mean trait that shows how dilution moves a file across a fixed line.

15.3.2 Entropy Worked Example in Full

Entropy is a measure of randomness in bytes. High entropy looks like packed, encrypted, or compressed bytes. Low entropy looks like plain code or text.

Shannon entropy for a byte source with symbols $i = 0, \dots, 255$ is:

$H = - i = 0 \sum 255 p_{i} lo g_{2} p_{i}$

Spoken form kept: entropy tells whether a file looks like malware or not. Here $p_{i} \in [0, 1]$ is the share of byte value $i$ in the file, $\sum_{i} p_{i} = 1$ , and $H$ is in bits per byte, from 0 to 8. High $H$ near 7 to 8 suggests packed or encrypted bytes. Mid $H$ near 5 suggests more normal mix. The log base is 2, which is the normal byte choice giving units of bits per byte; the lecture did not say the base aloud, so base 2 is fixed here to match byte practice. With base 2, a uniform mix over all 256 values gives $H = lo g_{2} 256 = 8$ , while a file with only one repeated byte gives $H = 0$ .

Build the bounds step by step so the range is not magic. If one byte value fills the whole file, say $p_{0} = 1$ and rest 0, then $H = - 1 \times lo g_{2} 1 = 0$ . If all 256 values share equally, $p_{i} = 1/256$ , then $H = - \sum_{i = 0}^{255} (1/256) lo g_{2} (1/256) = - 256 \times (1/256) \times (- 8) = 8$ . Real files sit between: plain text near 4 to 5, packed code near 7 to 8.

Setup: a defender uses mean file entropy as a trait with a line near 6. Above the line means suspect. Below means pass. The file has pockets of packed code with high local entropy.

Given: packed pockets push file $H$ to about 7.5. Goal of attacker as stated: bring $H$ down toward about 5 while keeping the payload runnable.

Worked dilution: from 7.5 toward 5 with live payload. Step 1: try compression. A learner asked if shrinking the file helps. Test it in mind. Packing 100 varied items into a tiny space removes visible structure. A 7-Zip archive at top level often shows higher entropy, not lower. So compressing the already packed file pushes $H$ up, not down. Result: $H$ stays near 7.5 or rises. This path harms the attacker. It also matches history. Attackers used packers, obfuscators, and encryption to hide static bytes. That hid bytes but raised entropy. More packing adds to the same problem.

Step 2: try dilution. Add plain benign code that has nothing to do with harm, such as hello-world prints, prime-number loops, or plain Java methods. Say the first file has $N = 10, 000$ bytes at $H \approx 7.5$ . Add $M = 30, 000$ bytes of plain code at $H \approx 4.5$ . Total bytes are $N + M = 40, 000$ . The pooled byte mix is now led by the plain part, with weight $M / (N + M) = 0.75$ on the low-entropy part. The pooled $H$ drops toward about 5.3. Numbers here are teaching values to show the pull, not lab measures. The payload still runs because the added code does not touch its logic. The detector sees mean $H \approx 5$ and passes the file. Final answer: trait flips from suspect to pass while payload stays live.

Sense-check: dilution works since entropy is a mean trait that moves bit by bit with added bytes. It does not snap like a hash. That is why padding pulls the mean while packing pushes it up.

Analogy kept in full. It is like thinning milk with water. The mix still looks like milk. Here the file is thinned with benign-looking bytes so much that an adulteration check misses the bad part. The attacker wins because the trait moved while acts stayed. Where the analogy breaks: milk thinned too far tastes watery to a human, but a mean-entropy check has no taste buds. It only sees the pooled number.

Exam note: This dilution walk is a classic applied number question, so practice each step with shares and logs stated. State $N$ , $M$ , start $H$ , added $H$ , pooled $H$ , and the live-payload check.

15.3.3 Boundary Points and Firewall Probing

A decision boundary is the line or surface where the choice flips. Points near it need only a tiny push to cross. Real traits often place files near that line. A small trait change then flips the label while acts stay whole and the executable still runs.

The same idea fits networks. Probe a firewall for open ports. Learn which traits the model leans on. Then shape those traits to pass. The model still sees only traits. It does not see intent. An attacker who learns that mean size carries high weight will shape sizes, not timing. One who learns that port counts matter will spread probes to look tame.

For a mean trait, let packets be $x_{1}, \dots, x_{n}$ in bytes and let mean size be:

$μ = \frac{1}{n} i = 1 \sum n x_{i}$

Spoken form: average packet size as a trait. Here $x_{i} \in R_{> 0}$ is size of packet $i$ , $n$ is count, and $μ$ is mean size. One huge $x_{n}$ can drag $μ$ far. That single move can flip a choice that leans on $μ$ .

Worked mean flip. Take $n = 20$ packets at $μ = 500$ bytes, so sum is $10, 000$ bytes. Add one huge packet of $10, 000$ bytes. New $n = 21$ . New sum $= 20, 000$ bytes. New $μ = 20, 000/21 \approx 952.4$ bytes. The mean almost doubles from one packet. If the suspect line sits near 700 bytes, the flow jumps from pass to suspect, or the reverse if the attacker shapes the move the other way. Final answer: one packet moves the mean by about 452 bytes. Sense-check: means have no guard against outliers, so a single large value drags the whole trait.

The guard follows from the math. A mean over a small window with no cap is fragile. Caps, medians, or trimmed means blunt the drag. That is why mature pipelines cap sizes and watch for single-packet jumps before trusting $μ$ .

Real-world: port probes, packet-size shaping, and junk-code padding all target the traits a model leans on most. Each trick picks the highest-weight trait and pushes it the shortest distance across the edge.

Recap: Near-edge points flip with tiny trait moves, in files and in flows alike. The bridge is goals: untargeted pushes need only cross anywhere, targeted pushes must land on a picked side.

15.3.4 Student Questions and Answers

Q: To beat an entropy check, should we shrink the file or pad it with sensible benign bytes, and does this relate to hashing tricks?

A: Shrinking by packing will raise entropy and hurt the attacker, since packed bytes look more random. Packing varied items into a tiny space removes structure, so a 7-Zip style top layer reads near 7.5 or higher. Padding with plain benign code thins the mix and can pull mean entropy from near 7 to 8 down toward about 5 while the payload still runs. Hashing tricks differ, since a hash changes fully with any byte flip, while entropy is a mean trait that moves bit by bit with added bytes. So pad to fool a mean, flip a byte to change a hash; the two traits need opposite moves.

15.4 Attack Goals and Single-Step Evasion

15.4.1 Untargeted Versus Targeted Goals

An untargeted attack seeks any wrong choice. The attacker does not care which wrong label appears. Call a harmful binary an email, a cat, a dog, or anything except malicious. The win is simply a wrong prediction.

Recall two model shapes from earlier sessions. A binary model answers benign or malicious. A probability model gives a score such as $p \in [0, 1]$ for malicious. In both shapes, untargeted success means the score lands on the wrong side. This is the most common shape in practice. Pad with NOPs, which are no-operation instructions that do nothing, or add benign-looking junk. The aim is to dodge, not to steer.

Mind-game analogy kept. In a login mind game, one player may only want to lock the other out, not to guess the true password. Havoc is the goal. Untargeted attacks share that spirit. Havoc, not control. The attacker wins when the guard says anything except the truth.

A targeted attack seeks a chosen wrong choice. Force the model to output a picked class. Example: camouflage command and control beaconing as normal HTTPS traffic. Here C2 means command and control, and beaconing means short periodic check-ins. The attacker must mimic the real class closely, not just tweak logs. That mimicry is much harder. It needs faithful timing, sizes, and headers. So targeted wins are rarer and costlier.

Let $y$ be the true label and $y_{t}$ a target label with $y_{t} \neq = y$ . Let $f (x)$ be the predicted label. Untargeted success is:

$f (x_{a d v}) \neq = y$

Targeted success is:

$f (x_{a d v}) = y_{t}$

Spoken form: wrong prediction for untargeted, and forced wrong class for targeted. Here $x_{a d v}$ is the crafted input. Targeted adds the extra demand that the wrong label equal a picked $y_{t}$ . In a binary case, the two goals look the same since only one wrong label exists. In a multi-class case with cats, dogs, mail, and malicious, untargeted has many winning outputs while targeted has only one.

A cost view makes the split stick. Untargeted search stops at the first crossing of any edge. Targeted search must keep walking until it reaches one named region, often crossing past nearby wrong regions it cannot accept. That longer walk needs more queries, tighter mimicry, and more live-payload care. So targeted costs more in time, queries, and risk of tripping monitors.

Real-world: junk padding to dodge a binary detector is untargeted, while shaping beaconing to read as normal HTTPS is targeted. The first pads until the score drops. The second must match HTTPS timing, sizes, and headers closely enough to read as that exact class.

Exam note: Be ready to label a story as untargeted or targeted and to state why targeted costs more. Name the win condition in symbols, then state the extra mimicry cost.

15.4.2 What Evasion Must Preserve

Evasion means changing a harmful input so it passes checks while its harmful acts stay live. Both halves matter. Pass plus dead payload is a loss. The ransomware must still encrypt. The thief must still steal. The beacon must still call home.

Simple story kept. Take a harmful file. Add junk. Dodge the check. Yet the file no longer runs. What was gained. Nothing. The attack fails its own purpose. So every evasion plan must test two things. Did it pass. Did it still act.

Scope: Function checks belong in the attack loop, not after it. Run the payload in a sandbox after each trait move. If execution breaks, drop that move even when the score looks good. A bypass that kills acts is a failed bypass.

The same rule holds for flows. Shape packet sizes to move the mean, but keep the channel alive. If the beacon stops calling home, the move is useless even when the detector says benign. Preserve-acts testing is what separates security evasion from image evasion, where a changed pixel still leaves a viewable image.

Recap: Pass plus live acts defines evasion success. The bridge is the cheapest way to get both: a single well-aimed push.

15.4.3 FGSM Formulation and Packet Average Demo

FGSM stands for Fast Gradient Sign Method. It is a single-step attack. It finds traits the model leans on most and nudges them once in the worst direction.

Sensitivity here means a tiny trait move flips the choice. Features are inputs to the model. So the attacker watches by trial which trait moves the score most. Then the attacker pushes that trait.

Let $J (t h e t a, x, y)$ be the training loss that is high when the model is wrong on $(x, y)$ . Let $\nabla_{x} J$ be its gradient with respect to $x$ , which points to the steepest rise in loss. Let $ϵ > 0$ be a small budget. FGSM crafts:

$x_{a d v} = x + ϵ \cdot sign (\nabla_{x} J (t h e t a, x, y))$

Spoken form kept: identify which features model is most sensitive to and flip with one step; do not worry too much about gradients, grasp the idea. Here $sign (\cdot)$ takes the sign per trait, so each trait moves by either $+ ϵ$ or $- ϵ$ in one go. $ϵ$ caps the size of the push per trait, which matches an $L_{\infty}$ bound of radius $ϵ$ ; no other bound was spelled out in words, so fix this reading: each trait moves at most $ϵ$ , total change stays in the box $∥ x_{a d v} - x ∥_{\infty} \leq ϵ$ . $x_{a d v}$ is the new input. Think of the gradient as the slope under a hiker's boots: the sign says which way is uphill for loss, and $ϵ$ says how long a step to take.

Why the sign and not the raw gradient? Two reasons. First, the sign spreads the budget evenly so no single trait hogs the push. Second, it keeps the move small in every trait, which is harder to spot than one huge spike. One step, all traits, fixed size per trait. That is the whole method in one line.

Worked demo with mean size. Setup: model uses mean packet size $μ$ and gap time as traits and leans on $μ$ . Given: normal $μ \approx 500$ bytes with $n = 20$ packets. Sum is $n μ = 10, 000$ bytes. Attacker sends one huge packet of $10, 000$ bytes. New $n = 21$ . New sum $= 20, 000$ bytes. New $μ = 20, 000/21 \approx 952.4$ bytes. The mean almost doubles from one packet. If the line for suspect sits between 500 and 952, the choice flips in milliseconds. Reported reach in class: about 70 to 85 percent evasion against undefended models. That range means most tries pass when no defense is in place. Final answer: $μ$ moves from 500 to about 952.4, a jump of about 452 bytes from one packet. Sense-check: a mean with no cap must move this way; the math leaves no other outcome.

Hands-on idea kept. Take the model built for the assignment. Find traits it leans on. Wear an attacker hat. Try to shift those traits in small legal ways and see if choices flip. You will learn more from that loop than from reading alone. List traits, move one trait a little, watch score and label, keep acts valid, and note which trait flips first.

Testing analogy kept. Writing code to find a mean should include negative tests such as zero denominator. If you test $n = 0$ and guard it, later tricky inputs cannot crash that path. Guard with a check such as if $n = 0$ return 0 or raise a handled flag before dividing. That guard is non-AI adversarial training. In ML terms, what-if samples plus fixes make the model stronger. This is a good red-team baseline: probe edges, add guards, re-test.

Scope: FGSM assumes the attacker can compute or guess the gradient direction. With full sight, the gradient comes straight from weights. With query-only sight, the attacker must guess it by probing, which costs more queries and is less exact. The formula stays the same; the cost to get the direction changes with sight.

Visual intuition helps. Draw loss on the y-axis and one trait on the x-axis. Mark the clean point on a slope. The gradient is the slope at that point. FGSM steps uphill by $ϵ$ in the sign direction. If that single step crosses the decision line, the attack wins at once. If not, a stronger loop is needed, which leads to the next section.

Real-world: single-packet mean flips and fast junk padding show how cheap single-step tricks can be. One packet or one padding block can move a mean trait enough to flip an undefended model.

Recap: FGSM pushes every trait once by $ϵ$ along the loss slope. The bridge is iteration: when one push falls short, many small pushes often still win.

15.4.4 Student Questions and Answers

Q: If we test our own assignment model, how should we start sensitivity probing without deep math?

A: List the traits, change one trait a little at a time, and watch the score $s$ and label. Keep the payload or task valid at each try. Note which trait flips the label with the smallest move. That trait is the sensitive one. Then think what guard, such as caps, windows, or extra traits, would blunt that move. Start with mean-type traits since they often flip first, then test counts and flags. Log each try as trait moved, size of move, score before and after, and whether acts stayed live.

15.5 Iterative Minimal and Query-Based Attacks

15.5.1 PGD: Many Small Steps

PGD stands for Projected Gradient Descent. It is the gold-standard iterative attack. It was set out by Madry and colleagues in 2018. Instead of one push, it takes many small refined steps, often 20 to 100. Each step moves a little and then projects back into the allowed range.

Why iterate when one push is cheaper? One push often stops short of the edge or overshoots into an invalid input. Many small steps feel the slope again after each move, stay inside the allowed ball, and land closer to the worst spot. Iteration trades compute for reach.

Purpose: find the strongest allowed tweak within a small budget when one FGSM step is not enough. Inputs are the clean input $x$ , true label $y$ , loss $J$ , step size $α > 0$ , budget $ϵ > 0$ , step count $T$ such as 20 to 100, and norm for the ball. Output is $x_{a d v}$ that fools the model while staying near $x$ .

Let $B_{ϵ} (x) = {z : ∥ z - x ∥ \leq ϵ}$ be the allowed ball around $x$ . Let $α > 0$ be the step size. Start at $x^{0} = x$ . For $t = 0, 1, \dots, T - 1$ :

$x^{t + 1} = Π_{B_{ϵ} (x)} (x^{t} + α \cdot sign (\nabla_{x} J (t h e t a, x^{t}, y)))$

Spoken form kept: take many small iterative steps and refine mods so evasion grows stronger. Here $T$ is step count such as 20 to 100, $Π$ is projection back into the ball, which clips any step that leaves the ball back to its edge. The norm $∥ \cdot ∥$ caps total change; the lecture did not say the norm aloud, so fix the common reading used with Madry PGD: $L_{\infty}$ with per-trait cap $ϵ$ unless stated otherwise, with $L_{2}$ as the other common choice. Reported reach: about 85 to 98 percent against undefended models, above the 70 to 85 percent of single-step tries.

Steps in plain order: start at the clean input, compute the loss slope at the current point, step by $α$ along the sign, project back into the ball, repeat $T$ times, keep the first point that flips the label with live acts. Projection is the key extra over FGSM. Without it, steps drift far and become easy to spot. With it, the search stays in the small box where tricks look tame.

Trace on a toy score. Say score $s = 2 x_{1} - x_{2}$ with threshold $t = 3$ for malicious. Clean $x^{0} = [2.0, 1.0]$ gives $s = 3.0$ , just malicious. Attacker wants benign ( $s < 3$ ) with $ϵ = 0.5$ per trait and $α = 0.2$ . Loss slope points toward lower $s$ , so sign says drop $x_{1}$ , raise $x_{2}$ . Step 1: $x^{1} = [1.8, 1.2]$ , $s = 2.4$ , already flipped. PGD would still refine to push $s$ lower within the ball, such as $x^{2} = [1.6, 1.4]$ with $s = 1.8$ . Final answer: flip in one refined step, hardened by further steps to 1.8. Sense-check: each step moves $s$ by $2 \times 0.2 + 1 \times 0.2 = 0.6$ , so the math tracks the score drop exactly.

Stop-sign story kept in full. A clean stop sign reads fine. Add black and white tapes and a self-driving car misses the sign and does not stop. The hard part is not that tape can fool. The hard parts are how many tapes, where to place them, why black and white and not red and white or plain white, and why that wall spot and not another. Those answers came from trials and study, not guesswork. PGD is the same idea in trait space. It searches where and how much to push. The tape color and spot are the trait picks; the count is the budget $ϵ$ .

Entropy lab kept as the cyber twin. Take a harmful file flagged as malicious. Add two benign instructions. Check entropy and score. Add three more. Check again. Keep adding plain code and checking if pooled entropy falls toward about 5. Stop when the label flips yet the file still runs. That loop of add and check is the iterative habit. Short videos and guides can fill math gaps beyond this sketch. This loop is PGD by hand: small add, re-score, stay valid, repeat.

Sight question kept. Does this attack change weights. It depends on sight and access. Goal is to fool the system. With a black box you cannot touch inner weights. You can only pad inputs again and again. With a local open copy, such as a fetched open model you host, you can tweak weights too. So separate attack type from sight level. Black-box tries in this part were high-level sketches of query-only shaping. Complexity is $T$ times one FGSM cost, so 20 to 100 times slower than one step but with higher reach.

Defense tip kept. Test defenses against PGD, not only against single-step tries. Single-step success does not mean true strength. Start there, but prove strength with the stronger loop. A guard that stops FGSM but falls to PGD with 20 steps is not strong.

The trap is to tune $α$ too large. Huge steps bounce across the edge and miss the worst spot. Small $α$ with more steps finds it. When to use PGD in red drills: use it as the baseline strong test. When to pick an alternative: use FGSM for a fast smoke test, use C and W when you need the smallest tweak rather than the strongest tweak in a fixed ball.

Real-world: stop-sign tape misses and entropy padding loops both show iterative search for placement and amount. Both needed trials to learn where and how much.

Exam note: Be ready to contrast one-step and many-step attacks by step count, evasion range, and cost. Quote FGSM 70 to 85 percent in one step versus PGD 85 to 98 percent in 20 to 100 steps at 20 to 100 times the cost.

15.5.2 C and W: Smallest Change and Physical Analogies

C and W stands for Carlini and Wagner, the researchers behind the method. The aim shifts from many steps to smallest change. Find the tiniest tweak that still flips the choice.

Why care about smallest rather than strongest? Large tapes are easy to spot and to counter. A tiny dot or thin line that still flips is much harder to spot. Defenders can filter gross changes. They struggle with minimal ones. That gap is why minimal-change attacks set the bar.

Purpose: find the least visible tweak that still fools the model. Inputs are clean $x$ , a fooling measure $f$ , trade weight $c > 0$ , and norm $p$ . Output is the smallest $δ$ that flips the choice while staying valid.

Let $δ$ be the tweak with $x_{a d v} = x + δ$ . Let $∥ δ ∥_{p}$ be its size. Let $f (\cdot)$ be a loss-like term that is small when the model is fooled and $c > 0$ a trade weight. The sketch is:

$δ min ∥ δ ∥_{p} + c \cdot f (x + δ)$

with $x + δ$ kept in valid range such as $[0, 1]^{n}$ for images. Spoken form kept: what is the smallest possible change for a flip, not count of steps or blind trials. Here $p$ picks the size measure such as 2 or infinity, and $c$ trades size against fooling strength. The lecture did not dictate exact objective terms in words, so fix this standard reading: $∥ δ ∥_{p}$ penalizes size, $f$ penalizes failure to fool, and larger $c$ pushes harder to fool at the cost of larger $δ$ . Search tunes $c$ to find the smallest fooling $δ$ .

Why this matters for defense. Large tapes are easy to spot and to counter. A tiny dot or thin line that still flips is much harder to spot. That is the scary part. Reported reach if no defense is planned: about 95 to 100 percent, near perfect. Cost: about 100 to 1000 times slower than PGD since search for the minimal tweak needs far more work, including tuning $c$ and refining $δ$ over many rounds.

Privacy analogies kept. News teams that shield victims often cover the full face, blur it, or at minimum place a black bar over the eyes of kids or victims in a story. No efficacy study was claimed for that habit. It is a redaction habit. Map views blur pedestrian eyes in street views, since eyes carry strong identity cues. To dodge crowd cameras, a small multicolour patch on a shirt was studied as a minimal trick that can wreck detection. A full mask also hides, but the patch aims for minimal. A see-through face mask was noted as a trick to warp face recognition. A small colour patch to dodge visual detection was shown with a ScienceDirect paper. Shown image described: a person reads as human to a common detector, yet with a small patch placed nearby the detection fails. Vision cases are used since they are easy to picture. Malware cases are harder to picture but share the same math of minimal tweak: a few padded bytes or imports that flip traits with almost no size change.

Minimal-change reading. Compare three wins on the same model. FGSM wins with a per-trait push of $ϵ = 0.1$ across all 1000 traits, so size is large in $L_{2}$ . PGD wins within the same ball but with a stronger flip. C and W wins with $∥ δ ∥_{2} = 0.02$ concentrated on 5 key traits. All three flip the label, but the C and W tweak is 5 times smaller and far less visible to a size filter. Final answer: smallest tweak wins the stealth prize even at 100 to 1000 times the search cost. Sense-check: size filters catch the first two but miss the third, which is why C and W sets the bar for strength claims.

Scope: Minimal does not mean valid by default. In security, the tiny $δ$ must still sit in safe spots that keep acts live, such as trailing bytes or dead code. A minimal pixel tweak always leaves a valid image, but a minimal byte tweak can break a binary. Validity checks stay mandatory.

Real-world: face masks, eye bars, map blurs, shirt patches, and detector patches all show minimal-change thinking in physical space. Each hides the key cue with the least cover.

Recap: C and W trades compute for stealth, finding the tiniest fooling tweak. The bridge is tooling: query-based and transfer paths that carry these tricks to targets with little direct touch.

15.5.3 Other Evasion Families and Tooling

ZOO, HopSkipJump, and transfer attacks round out the set at high level. Read them as extra paths, not as exam proofs.

Query-based attacks probe inputs and outputs to learn the edge. ZOO guesses gradients from scores by sampling near the point. HopSkipJump walks the edge with only labels, starting from a far flipped point and stepping closer while staying flipped. Transfer attacks build a local stand-in on similar data, craft tricks against the stand-in, then carry them to the real target with little or no direct touch. Hosting a stand-in on a public hub is one path noted. Less touch means less chance to trip monitors.

Separate two axes. Axis one is attack type such as FGSM, PGD, or C and W. Axis two is sight such as full access or query-only. FGSM with full access is a far more deadly mix than FGSM with no inside view, since weights guide the push. With no idea of the target model, the attacker must rely on blind trial and that is very hard. Attackers in the wild mix methods. They do not carry taxonomies. They chain theft plus evasion plus probing. Defenders use maps like OWASP frames to plan in a sane way, but attackers mix and match holes.

Tooling kept. The Adversarial Robustness Toolbox holds many of these attacks in code. If you know the target stack uses TensorFlow, you can pick matching attack modules. Like training, these modules have settings akin to hyperparams. You can try settings and payloads in trial loops to see which bypass holds. Feature work stays confusing until real projects force you to choose, clean, and test traits. Query-based probing plus local stand-ins is the habit to practice. A typical loop is: probe a few queries, fit a stand-in, craft offline, test one live carry-over, adjust settings, repeat.

Real-world: Adversarial Robustness Toolbox with TensorFlow modules and hyperparam-like attack settings, plus public-hub hosting of stand-in models, shows how lab tricks turn into repeatable red-team code. OWASP-style checklists turn those loops into planned coverage rather than random tries.

Recap: Query-based and transfer paths extend FGSM, PGD, and C and W to query-only targets. The bridge is defense: test against the strongest loop, not the cheapest one.

15.5.4 Student Questions and Answers

Q: With full access to weights, is evasion easier, and can theft plus evasion combine in real work?

A: Yes. Full access shows weights $t h e t a$ and gradients, so FGSM and PGD pushes are far more exact and more deadly. With query-only access the attacker must probe and guess, which is slower and harder. Real attackers chain methods, such as stealing a stand-in and then crafting evasion offline, so plan defenses that assume mixed and chained tricks, not single clean attacks. That chain is theft plus evasion: query to copy the edge, then craft without touching the target until the final carry-over.

Q: Does a backdoor need full control of training, and is poisoning hard without access to data?

A: Yes. To plant a reliable backdoor or poison labels at scale, the attacker needs a way to shape training files or labels or weights, such as hijacking the training path or slipping in rigged files. Without data or training access, that path is quite hard since the attacker cannot set the trigger-label link. That is why supply checks, hash tracking, and source trust matter. They close the training-access path that backdoors and poisoning need.

Q: Should we roll back to signature methods, or mix old and new defenses?

A: Always mix. No tool fixes a problem without raising new ones. Use signatures plus ML plus monitoring. Signatures catch known bytes fast, ML catches novel traits, monitoring catches drift. Train models with both normal and tricky samples, keep red teaming the pipeline, and track drift in traits and scores over time. Rollback alone gives up novel-catch power; ML alone gives up fast known-catch power.

15.6 Strong Training and Layered Defenses

15.6.1 Adversarial Training and Its Trade-Offs

Adversarial training means training on both normal and tricky samples so the model gains a form of immunity. You attack the model during training, see where it breaks, fix, and repeat. This mirrors red teaming for systems. You test to break, then you harden.

Why train on attacks instead of just more clean data? More clean data sharpens the bulk but leaves the edge bare. Tricky samples fill the edge. The model learns that near-edge points with small pushes still belong to the true class. That edge teaching is what clean data alone cannot give.

Let $D$ be clean samples and $D_{a d v}$ be crafted tricky samples, such as PGD outputs within budget $ϵ$ . Training minimizes loss on the union:

$t h e t a min (x, y) \in D \sum J (t h e t a, x, y) + (x_{a d v}, y) \in D_{a d v} \sum J (t h e t a, x_{a d v}, y)$

Spoken form kept: train on both normal and adversarial samples to build immunity; attack during training and check if resistant. Here $t h e t a$ holds weights, $J$ is loss, and the second sum forces correct acts on tricky inputs too. In practice the tricky set is rebuilt each round as weights change, so the min wraps an inner max that crafts fresh $x_{a d v}$ for the current $t h e t a$ .

A step-by-step loop makes it concrete. Round 1: train on $D$ only. Round 2: craft $D_{a d v}$ with PGD against current $t h e t a$ . Round 3: retrain on $D \cup D_{a d v}$ . Round 4: re-craft harder $D_{a d v}$ since the edge moved. Repeat until attacked accuracy stops rising. Each round costs extra forward and backward passes for crafting, which is why compute rises.

Worked trade-off numbers kept with no rounding. Without defense, normal accuracy near 95.2 percent can fall to near 0.3 percent under PGD, which means nearly all attacks pass. The drop is $95.2 - 0.3 = 94.9$ points, a near-total collapse. With adversarial training, normal accuracy may dip a little since defense layers add load, but accuracy under attack rises by tens of points. Costs: about 2 to 5 times training compute, and about 1 to 8 percent drop on normal inputs in the cases noted, with more false positives. Final answer: trade a 1 to 8 point clean dip plus 2 to 5 times compute for rescue from 0.3 percent attacked accuracy. Sense-check: the clean dip is small next to the 94.9-point collapse it prevents, but it still matters when false positives block good files.

The juggle is between clean score and attacked score. A tighter defense that resists larger $ϵ$ often drops more clean points and fires more false alarms. Teams must pick the point that fits risk: a laptop detector may accept more false positives, a mail gateway may not. High clean score plus high attacked score at once is hard. That tension persists in production.

Scope: Adversarial training guards the budgets and attacks it saw. A model trained against $L_{\infty}$ PGD at $ϵ = 0.1$ may still fall to C and W minimal tweaks or to larger budgets. State the budget and attack used in training, and re-test against stronger loops before claiming strength.

Real-world: production detectors that add adversarial training accept higher compute and a small clean-score dip to avoid collapse under PGD. They track both numbers on one board and block release when attacked accuracy falls short.

Exam note: Learn the 95.2 to 0.3 collapse story and the 2 to 5 times compute plus 1 to 8 percent clean dip as the core trade-off numbers. Quote the pair, not one side alone.

15.6.2 Certified Defenses Smoothing Preprocessing Ensembles

Certified defenses aim for math-backed promises against tricks within a range. They are theory-first, not proof by live trial. Think formal proof versus live pentest. Formal proof reasons on paper. Pentest tries live. Both help. Proof alone is not enough.

Anecdote kept without names. During doctoral work, a line of research on protocol safety stood out. It builds math and tools to argue if a handshake protocol such as TLS or SSL is safe. Later work proved newer handshakes such as TLS 1.3 and MLS in that style. That program ran for 20 to 30 years and built tools where you write steps and claims and get a safety argument. It feels deep. Yet a classic line often linked to Donald Knuth warns in spirit that a proof of an algorithm is not proof of no bugs in the build. Paper proof is on paper. You must still test the running system. Proof papers plus live tries give joint trust; either alone leaves a gap.

Other layers include randomized smoothing, which votes over noisy copies of an input. Take many jittered variants, predict each, and take the majority. Input preprocessing trims precision or windows traits to blunt tiny pushes. Example: cap sizes, use a small window for mean, or test how small input shifts move outputs and then damp those moves. Ensembles vote across models A, B, and C. Some models break easily. Some hold. A vote can survive one weak member.

Let $x + η_{k}$ be noisy copies with noise $η_{k}$ for $k = 1, \dots, K$ . Let $f (\cdot)$ give labels. Smoothed choice is:

$y^= mode {f (x + η_{1}), \dots, f (x + η_{K})}$

Spoken form: make predictions on randomized input variations and vote. Here $mode$ is the most common vote and $K$ is copy count. Noise $η_{k}$ is often Gaussian with scale $σ$ ; larger $σ$ smooths more but drops clean accuracy. $K = 100$ votes with 70 for benign gives $y^=$ benign even when the single clean $x$ reads as malicious under a tiny push.

Each layer has a clear job. Smoothing blunts tiny pushes since one push rarely flips most noisy copies at once. Preprocessing caps the drag from outliers: cap packet sizes at 1500 bytes, use a median over a short window, or quantize entropy to coarse bins. Ensembles spread risk since one weak member rarely swings the majority. None holds alone, but each raises attacker cost.

Worked smoothing vote. Take $K = 11$ noisy copies of a borderline file trait. Single model on clean $x$ says malicious by a hair. Noisy votes come back as benign 7 times and malicious 4 times. Mode is benign, so $y^=$ benign. A tiny push that flips the single clean read must now flip at least 6 of 11 noisy reads to swing the vote. Final answer: majority holds benign 7 to 4 despite the single-read flip. Sense-check: the attacker needs a push large enough to move the bulk of the noise cloud, not just one point.

Scope: Smoothing needs enough $K$ to steady the vote and the right $σ$ to match the threat. Too small a $σ$ leaves tiny pushes intact. Too large a $σ$ washes out true traits. Preprocessing that is too coarse drops clean accuracy. Tune all three on both clean and attacked sets.

Real-world: smoothing votes, precision caps, and A-B-C model votes appear in hardened vision and malware pipelines. Vendors report both clean and smoothed-attacked numbers so buyers see the trade.

Recap: Certified promises, smoothing votes, input caps, and ensembles each blunt a slice of the attack space. The bridge is stacking them so gaps in one are covered by the next.

15.6.3 Defense in Depth and Hybrid Thinking

No single layer holds alone. Use defense in depth, which means stacked layers that cover each other. Stack adversarial training plus input checks plus drift watch plus fast incident response. Guard models the way you guard IT systems with steady testing.

Hybrid design also holds. Keep signatures with ML. Signatures catch known bytes fast. ML catches novel traits. Monitoring catches drift. Human review catches edge cases. Drift was raised in discussion and affirmed. Attacks shift. IT setups shift. Business mix shifts. Static models go stale. You must retrain and redeploy on a cadence. A quarterly retrain with fresh tricky samples beats a one-time hard push that rots in six months.

Quote-idea kept in plain words. No tool fixes a problem without raising fresh problems. The prior sessions showed ML solving real defense tasks. This session shows the fresh problems ML brings. Both views are true at once. The practical answer is not to drop ML but to pair it: signatures for known, ML for novel, smoothing and caps for tiny pushes, ensembles for single-model breaks, drift boards for shift, playbooks for fast response.

Stack walk on one file. A padded ransomware file arrives. Signatures miss since bytes are new. Input caps trim the huge trailing pad before scoring. The ML model trained with tricky samples still flags it as suspect. The suspicious-class route sends it to an analyst. The drift board logs the new padding shape for the next retrain. Final answer: four layers where one alone would have missed. Sense-check: each layer covers the miss of the prior one, which is the definition of depth.

Real-world: shops that pair signature lists with ML scores plus drift boards plus playbooks show depth rather than reliance on one model. Their release gates check clean accuracy, PGD accuracy, and drift alerts together.

Recap: Depth plus hybrid pairing plus retrain cadence is the durable posture. The bridge is detection: even with depth, flag odd inputs before they cause harm.

15.6.4 Student Questions and Answers

Q: How do we weigh clean accuracy against attacked accuracy when adding defenses?

A: Track both numbers on the same board. Note clean accuracy near 95.2 percent and attacked accuracy near 0.3 percent under PGD without defense. Then re-test with adversarial training and note the small clean dip against the large attacked gain, plus the 2 to 5 times compute cost and extra false positives. Ship the point that holds attacked accuracy high enough for the risk at hand. If false positives block real work, tighten caps or add human review rather than dropping the defense.

15.7 Detecting Adversarial Inputs Security Constraints Metrics Standards

15.7.1 Statistical and Architectural Detection

Detection tries to flag tricky inputs before they cause harm. Five patterns recur at high level. None is a full fix alone. Each raises cost or buys time for human review.

Density estimation per sample asks what normal looks like and flags odd ones. Spoken form: see what normal inputs look like and flag abnormal inputs. Take mean packet size. A huge packet that wrecks the mean is odd. Flag it. Take PE traits, where PE means portable executable file shape. High entropy in a PE part plus low import count, where import count means number of library calls listed, is odd. Flag it. These are tiny detectors for odd inputs.

Batch monitoring compares fresh batches to past baselines. A shift in submit patterns can mean probing. An attacker may send odd probes to learn the edge. This path is prone to false alarms. Worse, a detector for tricky inputs is itself a model that can be tricked, which adds a loop. So batch boards need their own guards: capped inputs, signed baselines, and analyst review before auto-block.

Augmented training adds a spare suspicious class to outputs. Train on clean plus tricky samples. Route suspicious outputs to human review. Spoken form: intentionally add a suspicious class and send those cases to analysts. It plants a form of deception inside the model to catch what slips past the main line. The spare class gives the model a place to put near-edge doubts instead of forcing a benign or malicious call.

Autoencoding and feature-space analysis look inside. An autoencoder is a net that learns to rebuild normal inputs through a narrow bottleneck and flags inputs that rebuild badly. Feature-space checks watch inner activations. Tricky inputs often cause odd activation patterns. Track those patterns to spot them. A high rebuild error or a strange activation vector is a flag even when the final label looks benign.

Fuzzing analogy kept. In non-ML testing, fuzzing sends random inputs to REST APIs to see if the app holds. The same spirit applies here. Send varied inputs and see if the model holds. Random trait jitters, padded variants, and edge-case files form a cheap fuzz suite that often finds fragile spots before attackers do.

High-level compare kept. Per-sample density, batch watch, augmented training, autoencoding, and feature checks each help to varying degrees. What helps most is training plus monitoring together. A model trained to withstand plus watched in prod beats a lone trick detector. Pair a strong base (adversarial training) with live eyes (density plus batch plus activation watch) and a human path (suspicious-class routing).

Let $p (x)$ be a density score for normal inputs and $τ$ a threshold. Flag if:

$p (x) < τ$

Spoken form: low likelihood under normal model means odd. Here $x$ is the input, $p (x)$ is its normal-mass score, and $τ \in R_{> 0}$ sets how strict the flag is. Raise $τ$ to catch more odd inputs at the cost of more false alarms. Lower $τ$ to cut alarms at the cost of more misses.

Worked flag. Model normal packet means near 500 bytes with spread 100 bytes. Set $τ$ so means above 900 flag as odd. A flow with $μ \approx 952.4$ from one huge packet gives low $p (x)$ since 952 sits more than four spreads above normal. Check: $p (x) < τ$ holds, so flag it. A PE part with entropy 7.6 plus import count 2 also gives low $p (x)$ under a joint normal model, so flag it too. Final answer: both odd inputs flag while normal flows near 500 pass. Sense-check: flags fire on distance from normal, not on the final label, so they catch pre-flip probes as well as full flips.

Scope: Density flags need a fresh normal model. After drift, old $p (x)$ misfires on new normal. Retrain the normal model on a cadence and track flag rates. A sudden jump in flag rate often means drift or probing, not a broken detector.

Real-world: packet-size flags, PE entropy plus import-count flags, batch drift boards, suspicious-class routing to analysts, and activation watches all appear in mature pipelines. Each is a small guard that feeds one incident queue.

Recap: Five detection patterns flag odd inputs by per-sample, batch, spare-class, rebuild, and inner-activation views. The bridge is security limits: vision-style flags need extra care to stay valid on files and flows.

15.7.2 Security-Specific Limits and Preserving Tricks

Vision tricks do not move as-is to security. Three gaps matter. Each gap forces extra validity work that image attacks skip.

First, acts must stay live. In vision, a tape on a sign still leaves a sign image. In security, a tweak that breaks the program kills the attack. If dummy bytes stop execution, the bypass is useless. So the attacker must keep program logic whole. Every candidate tweak needs a sandbox run before it counts.

Second, traits are discrete. Pixels form a smooth field where tiny shifts are easy. File traits are chunky. Counts, flags, and table entries jump in steps. Format rules add walls. Windows executables, Android packages, and ELF files each obey strict shapes. So pushes must sit in safe spots only, such as trailing bytes, dead code, or manifest fields that the loader ignores.

Third, detection is layered. One model rarely stands alone. Stacks of parsers, lists, models, and rules watch together. So the attacker must pass many gates, not one. A trick that fools the ML score may still trip a parser or a signature. Full bypass needs a pass at every gate.

Preserving tricks kept from attacker view. Manifest tricks touch Android package manifests, not code. A manifest lists permissions. Some look risky such as internet or send SMS. Some look tame such as flashlight or vibrate. Add tame permissions you do not need. The detector sees tame traits and softens. Code acts do not change since code was untouched. Dead-code injection adds good code to a harmful file, plus spaces, trailing bytes, or import-table padding. PE mods add spaces or extra data at file level while keeping run logic. All these aim to move traits yet keep acts. The shared math is the same as dilution: shift the trait vector with inert bytes while payload logic stays fixed.

Worked preserving triple kept. Practice numbers kept: about 83.4 percent evasion rate, about 6.2 mods per sample, about 97.1 percent acts preserved. That triple shows high pass with few touches and live payloads. Read it as: out of 1000 tries, about 834 fool the model; each win needs about 6 small mods on average; about 971 of 1000 wins still run. Final answer: high pass, few touches, live payloads together. Sense-check: few mods plus live acts is the scary mix since size filters miss small moves and sandbox checks pass live files.

Real-world: manifest permission padding and PE trailing-byte padding are classic live-preserving tricks in Android and Windows malware work. Defenders answer with validity-aware training that includes such padded samples and with parsers that strip or flag inert regions before scoring.

Recap: Live acts, discrete traits, and layered gates make security evasion harder than pixel tweaks but still feasible with manifest and PE preserving tricks. The bridge is measurement: judge strength with a shared triple, not clean accuracy alone.

15.7.3 Metrics Benchmarks and Worked Numbers

Three gauges judge strength under attack. Track them as a triple on one board. One alone misleads.

Attack success rate is the share of tries that fool the model. Let $N$ be tries and $N_{s}$ wins for the attacker:

$ASR = \frac{N _{s}}{N}$

Spoken form: share of attacks that succeed. High ASR means weak defense. If $N = 200$ and $N_{s} = 170$ , then $ASR = 170/200 = 0.85$ , or 85 percent.

Robust accuracy is the share still right under attack. Let $N_{c}^{a d v}$ be count right under attack:

$RobustAcc = \frac{N _{c}^{a d v}}{N}$

Spoken form: how model acts under attack conditions. Low RobustAcc means the model collapses when pushed. If $N = 200$ and $N_{c}^{a d v} = 30$ , then $RobustAcc = 30/200 = 0.15$ , or 15 percent. Note that $RobustAcc \approx 1 - ASR$ when each try is either a win for attacker or a correct hold, but the two can differ when rejects or suspicious routes count apart.

Modification size is how much the attacker had to change. Small change that still wins is more scary. Large change is easier to spot. This is the C and W lens. How much input had to move. Less move with a win means worse news. Report size in the same norm used for the threat, such as $L_{\infty}$ per-trait cap or $L_{2}$ total length, plus validity notes such as mods per sample.

Worked slide-down kept. Benchmarks exist for image tasks and for malware tasks to compare strength across teams. Worked image-classifier numbers kept: normal inputs about 87.3 percent right with adversarial training in that demo; FGSM still about 81 percent; PGD with 20 steps about 53.7 percent; C and W about 42.1 percent. The drop means the classifier mislabels far more as pushes grow smarter. Steps: $87.3 \to 81$ is a 6.3-point dip for one step, $81 \to 53.7$ is a 27.3-point fall for 20 refined steps, $53.7 \to 42.1$ is a further 11.6-point fall for minimal search. Final answer: stronger search pushes robust accuracy from 87.3 down to 42.1. Sense-check: each stronger method should lower the number; any rise would mean a test error. Now map that to cyber. If malware or intrusion models fell that far under C and W, they would be near useless in prod. That is why C and W sets the bar.

Real-world: published image and malware strength boards let buyers compare vendors on attacked accuracy, not only clean accuracy. Ask for the triple (ASR, RobustAcc, size) at named budgets before buying.

Exam note: Practice ASR plus RobustAcc plus size as a triple, and be ready to read the 87.3, 81, 53.7, 42.1 slide-down as growing attacker strength. Show the division for ASR and RobustAcc with $N$ stated.

15.7.4 Standards Frontiers and Closing Questions

Standards give guardrails for AI risk. The NIST AI Risk Management Framework helps teams assess AI systems, map attack types to principles, and run risk reviews. Many customers now ask for that shape of review as AI work spreads. A robustness standard numbered S24029 in discussion covers threat modeling, strength testing, and docs so models can hold ground under attack. ATLAS checklists and links give hands-on practices. Treat this part as pointers for reading, not as proofs to memorize.

Frontiers noted for next steps. Adaptive attacks tune to your exact defense, not generic spray. One C and W paper broke about 10 prior defenses in one go. AutoAttack bundles strong tries for fair tests. Generative AI safety adds prompt injection and jailbreaks as a fresh surface. The next iteration of this course plans to dig deeper there as industry moves that way. Concept drift means attacks, IT setups, and business mix all shift, so static models rot and need retrain and redeploy. Federated learning safety adds poisoning and Byzantine acts, where Byzantine means faulty or hostile peers in joint training. Those were named as advanced paths to browse after tests.

Q: Is the final test only on post-midterm sessions, what marks shape will it take, and is it open book?

A: Yes. Scope is post-midterm sessions from session 9 onward to now, not the full course. Shape is 40 marks with 2-mark, 3-mark, and 5-mark questions, four of each, twelve questions in all, where 2 times 4 is 8 plus 3 times 4 is 12 plus 5 times 4 is 20 for 40. It is open book with printed course material allowed from the uploaded set. Most marks come from class content, with a small share just beyond class to reward extra reading, mixing direct and applied questions. For this lecture, drill the post-midterm applied set: four-mode matching, entropy dilution steps, FGSM versus PGD versus C and W, the 95.2 to 0.3 collapse, and the ASR plus RobustAcc plus size triple.

Q: How should we pick a six-month project while working full time, and can webinar demos seed ideas?

A: First try to align with your day job, such as risk-based login work in IT safety or a banking safety need, so effort helps both work and study. If day work has no safety link, grow a webinar demo into a full build, such as malware detection, entropy checks, trait work, or attack drills from shared code repos, but do not submit a copy as your project. Keep scope small and doable in six months with job and family load, so you can show it in a portfolio. Ask for open problem lists if the program shares them, and route safety topics to the right guide where the process allows.

Q: Can we get the topic list for the next iteration focused more on generative safety, and when will assignment marks post?

A: The next-iteration list is still under review, with more generative safety planned as industry shifts that way, so mail after the course for a shareable copy. Assignment marks are compiled in a sheet and then moved to the portal one by one, with posting aimed within about two days given the 24th close date for term marks. Use the wait to drill applied answers and to line up a small project scope.

Exam Guidance Summary

Scope is post-midterm sessions from session 9 onward to now, not the full course. Next session is a full-course revision for the final test. Focus revision on sessions 9 through 15.
Shape is 40 marks. There are 2-mark, 3-mark, and 5-mark questions, four of each. That is twelve questions in all. 2 times 4 is 8. 3 times 4 is 12. 5 times 4 is 20. Sum is 40. Plan time by marks: short definitions for 2-mark, worked steps for 3-mark, full stories with numbers for 5-mark.
Open book with printed course material allowed from the uploaded set. Bring clean prints and know where each topic sits so you can apply fast. Tab the four-mode map, the entropy dilution walk, the FGSM versus PGD versus C and W table, and the metric triple.
Most marks, about 90 to 95 percent, come from class content. A small share, about 5 to 10 percent, sits just beyond class to reward curiosity and extra reading. Expect both direct and applied questions that ask you to use ideas, not only repeat them.
High-value applied drills: map a story to evasion versus poisoning versus stealing versus backdoor; label untargeted versus targeted and state cost; walk entropy dilution step by step with $N$ , $M$ , and pooled $H$ ; contrast FGSM versus PGD versus C and W by steps and reach (70 to 85 percent, 85 to 98 percent, 95 to 100 percent); quote the 95.2 to 0.3 collapse and the 2 to 5 times compute plus 1 to 8 percent clean dip; read ASR plus RobustAcc plus size as a triple with $N$ stated; list preserving tricks that keep acts live such as manifest padding and PE trailing bytes.
Study habits: grasp ideas, read widely, write your best. Do not seek lenient marks or press on re-evaluation notes. Marks fade. Ideas stay. For working learners with little time, drill one worked number per concept rather than re-reading all prose.
Exam note: bring printed material, practice short applied answers, and keep formulas with symbol meanings ready for fast use. State symbols $t h e t a$ , $x$ , $y$ , $ϵ$ , $μ$ , $H$ , $p (x)$ , $τ$ , $ASR$ , and $RobustAcc$ with one line each.

Key Industry Applications

Real-world: anti-malware and intrusion detectors in production face evasion, poisoning, stealing, and backdoors, so model safety is product safety. Release gates now check clean plus attacked accuracy together.
Real-world: Hugging Face spreads useful models and also rigged re-uploads with planted weight triggers, so hashing, source trust, and pre-use tricky testing matter. Pin hashes and re-test any fetched weight file before deploy.
Real-world: EMBER-based file detectors and packet-count network detectors live or die by trait design, dilution padding, and mean-shift tricks. Cap sizes, watch single-packet jumps, and add per-slice tests for packed sections.
Real-world: stop-sign tape misses, eye-bar redaction, map-view eye blurs, shirt patches, and see-through masks show minimal-change evasion in physical space that maps to C and W thinking. Small well-placed tweaks beat large random ones.
Real-world: Adversarial Robustness Toolbox with TensorFlow modules plus hyperparam-like attack settings gives repeatable red-team code for FGSM, PGD, and more. Teams loop over settings and payloads to find holds that survive the strongest loop.
Real-world: query-based probing plus local stand-in transfer hosted on public hubs shows how stealing plus evasion chain in the wild. Rate limits, minimal outputs, and probe monitoring blunt the copy step.
Real-world: production stacks pair signatures with ML scores, smoothing votes, A-B-C ensemble votes, suspicious-class routing to analysts, batch drift boards, and incident playbooks for depth. No single layer holds alone.
Real-world: NIST AI Risk Management Framework reviews, S24029 strength testing and docs, and ATLAS checklists guide buyer and builder risk work. Buyers ask for the ASR plus RobustAcc plus size triple at named budgets.
Real-world: banking safety needs, risk-based login, malware and entropy demo repos, and trait-work pipelines seed Larsen-scale six-month projects that fit job and family load. Align scope with day-job data where possible and keep builds small enough to finish and show.

AMTCS Lecture 15 notes · Adversarial Machine Learning for Security

AI & ML Techniques for Cyber Security· postgraduate· 2026-09-06

Sections Breakdown

115.1 ML as Security Infrastructure and Four Failure Modes

ML sits on blocking path so model failure is defense failure; four modes with distinct access needs plus silent failure motive for adversarial testing

215.2 Threat Taxonomy and Kill Chain for ML Systems

Attacker sight aims and lifecycle stages plus 127-page taxonomy ATLAS matrix and ML kill chain from recon to impact

315.3 How Models See Features and Why Small Shifts Fool Them

Trait-vector view with linear score and threshold plus full entropy dilution and mean-shift worked numbers

415.4 Attack Goals and Single-Step Evasion

Untargeted versus targeted goals with preserve-acts rule plus FGSM single-step form and packet mean worked flip

515.5 Iterative Minimal and Query-Based Attacks

PGD iterative projection and C and W minimal-tweak search plus query-based transfer families and ART tooling with chained theft-evasion habit

615.6 Strong Training and Layered Defenses

Adversarial training loop with collapse trade numbers plus certified smoothing preprocessing ensembles stacked as depth with hybrid retrain habit

715.7 Detecting Adversarial Inputs Security Constraints Metrics Standards

Detection five-pattern set with density flag plus preserving manifest PE tricks plus metric triple benchmarks plus standards and closing exam project guidance

8Exam Guidance Summary

Exam scope shape open-book rules and high-value applied drills with trade numbers

9Key Industry Applications

Production guards from hash pins to drift boards mapped to each attack family

Postgraduate students in AI and ML Techniques for Cyber Security

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

ML as Security Infrastructure and Four Failure Modes

Must-know: Four modes: evasion, poisoning, stealing, backdoor; silent failure means boards stay green while harm passes

⚠️ Top pitfall: Reading high clean accuracy as safety; mean hides aimed slice

Self-check: Match a story (modified binary as safe, ransomware labeled benign, 100 queries over weeks, trigger bytes as benign) to its mode and access need

Connects to: 15.2, 15.3, 15.6

Threat Taxonomy and Kill Chain for ML Systems

Must-know: Sight (white/black/gray) x aims (availability integrity privacy misuse) x stages (gather to inference); ATLAS is ML matrix in broader ATT and CK family

$f (x; t h e t a) with white box seeing x, t h e t a; black box seeing only (x, f (x; t h e t a))$

⚠️ Top pitfall: Writing gray box alone without naming the known slice; hiding weights treated as full defense

Self-check: Place a story by sight plus aim plus stage and name the guard that fits

Connects to: 15.1, 15.3, 15.5

How Models See Features and Why Small Shifts Fool Them

Must-know: Models read traits not acts; entropy dilution pulls 7.5 toward 5; one huge packet can double mean

$H = - i = 0 \sum 255 p_{i} lo g_{2} p_{i}, μ = \frac{1}{n} i = 1 \sum n x_{i}$

⚠️ Top pitfall: Packing to lower entropy; mixing hash snap with mean pull

Self-check: Given N=10000 at 7.5 plus M=30000 at 4.5, state pooled H and why payload stays live

Connects to: 15.1, 15.2, 15.4

Attack Goals and Single-Step Evasion

Must-know: Untargeted any wrong vs targeted picked wrong; FGSM one push of epsilon per trait; mean 500 to 952.4 demo; 70 to 85 percent reach undefended

$x_{a d v} = x + ϵ \cdot sign (\nabla_{x} J (t h e t a, x, y)), f (x_{a d v}) \neq = y vs f (x_{a d v}) = y_{t}$

⚠️ Top pitfall: Treating pass without live acts as a win; mixing sight cost with formula

Self-check: Label junk padding vs HTTPS-shaped beaconing as untargeted or targeted and state cost gap

Connects to: 15.3, 15.5

Iterative Minimal and Query-Based Attacks

Must-know: PGD 20-100 steps 85-98 percent Madry; C and W minimal tweak 95-100 percent at 100-1000x cost; ZOO HopSkipJump transfer plus ART TensorFlow tooling

$x^{t + 1} = Π_{B_{ϵ} (x)} (x^{t} + α \cdot sign (\nabla_{x} J)), δ min ∥ δ ∥_{p} + c \cdot f (x + δ)$

⚠️ Top pitfall: Testing only FGSM and claiming strength; mixing attack type with sight level

Self-check: Contrast FGSM vs PGD vs C and W by steps reach and cost; separate attack type from sight

Connects to: 15.4, 15.6, 15.7

Strong Training and Layered Defenses

Must-know: Adversarial training union loss; 95.2 to 0.3 collapse; 2-5x compute 1-8 percent dip; smoothing vote plus caps ensembles depth

$t h e t a min [D \sum J + D_{a d v} \sum J], y^= mode {f (x + η_{k})}$

⚠️ Top pitfall: Claiming strength from one budget or one attack; chasing clean score alone

Self-check: Quote collapse and cost triple and state why proof alone is not enough

Connects to: 15.4, 15.5, 15.7

Detecting Adversarial Inputs Security Constraints Metrics Standards

Must-know: Five detectors plus three security gaps plus ASR RobustAcc size triple; 87.3 81 53.7 42.1 slide and 83.4 6.2 97.1 preserving triple; NIST ATLAS S24029 pointers

$p (x) < τ, ASR = N_{s} / N, RobustAcc = N_{c}^{a d v} / N$

⚠️ Top pitfall: Tracking clean accuracy alone; treating vision tweaks as directly valid on binaries

Self-check: Compute ASR and RobustAcc from N and wins and read slide-down as stronger search

Connects to: 15.5, 15.6

Previous LectureLecture 14 · Adversarial Machine Learning Fundamentals

Next LectureLecture 01 · Introduction to Artificial Intelligence

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.