Quality Assurance and System Quality for Machine Learning

12.1 OOP Applicability in Agentic AI

Object-Oriented Programming (OOP) concepts — especially encapsulation, composition, and polymorphism — apply directly when building agentic AI systems. An agentic AI system is a system in which a large language model (LLM) performs a multi-step task by planning, using tools, and checking its own work, rather than answering a single prompt. The lecture motivates these concepts with a concrete analogy: a Hollywood film production crew.

12.1.1 The Monolithic Procedure Problem

Consider a movie set where the director does everything: holds the cameras, applies the actor's makeup, writes the script line by line, adjusts the lighting, and sweeps the floor. This is absurd — if the lighting breaks, the director stops directing to change a bulb. The situation is fragile and ambiguous: nobody can tell whether the movie fails because of the script, the camera work, or the lighting.

The same problem occurs in software when a monolithic procedure script runs and dozens of if-else statements guide an LLM through a complex workflow. Such a system is highly fragile and impossible to scale. There is no clear way to identify which part to scale, which performance to measure, or which piece of code to fix.

12.1.2 The OOP Crew Approach

In the collaborative (OOP) approach, the production crew is organized so that every crew member — actors, writers, cinematographer — has their own defined role, name, paycheck, and specialist assignment. Each member operates in their own bubble. The same idea underlies the four classic OOP principles: encapsulation, abstraction, inheritance, and polymorphism.

Encapsulation: Each crew member has its own set of tools and duties, keeping track of its own responsibilities independently. Like a capsule, each bubble contains its own scope. In code, encapsulation means a class hides its internal details from the outside — the rest of the system sees only the interface (the methods and attributes you choose to expose), never the wiring underneath.

Composition: The executive producer merely tells everyone to execute. Each member knows their job and performs it. The overall workflow is composed of these independent roles working together. In code, composition builds larger behavior by combining smaller, independent objects — the workflow does not contain the logic of each step, it merely wires the steps together.

Polymorphism: If a sound engineer needs to be added mid-production, there is no need to shuffle the entire group. The sound engineer plugs in wherever needed, and the existing structure adapts. In code, polymorphism (from Greek poly = many, morph = forms) means many different classes share the same interface: same method name, different behavior.

12.1.3 Agentic AI Example: Paper Validation Workflow

A concrete code demo — the paper validation system from earlier sessions — illustrates these OOP concepts in agentic AI. The system uses:

• An abstract base agent class with variables name (string) and model (string), which encapsulates agent identity.
• Two agents that inherit from the base: a validation agent and a summary agent. Both implement an execute method (returning a string), demonstrating inheritance and polymorphism — one abstract method splits into two different behaviors.
• The validation agent checks paper length and keywords.
• The summary agent synthesizes the abstract, producing a summary in about 200–300 words.
• Both agents feed into a research paper workflow — a composition that chains validation then summarization in sequence.

This single example shows all five OOP concepts: encapsulation (each agent owns its state), abstraction (the base class defines the interface), polymorphism (one execute method, two behaviors), inheritance (agents extend the base), and composition (the workflow composes agents into a pipeline).

12.2 Quality Assurance and System Quality

12.2.1 Why Quality Assurance Matters

Quality assurance ensures that what is being built matches customer expectations and delivered requirements. It measures how much a system deviates from what was promised. Before going to production, a bug-free application must be verified so that clients do not face issues while conducting business.

The eventual goal is to land the system into production, monitor it there, and ensure everything works well with production data. For this, understanding quality assurance and system quality is essential.

Quality assurance (QA) broadly refers to all activities that evaluate whether a system meets its requirements and whether it meets the needs of its users. Three properties are in play, and all three must be verified:

• Functional correctness — does the software produce the expected output for the given input?
• Operational efficiency — does it run fast enough, and within its resource budget?
• Usefulness — does an end user actually get value from it?

Why this matters for ML specifically: a traditional application's quality is dominated by code correctness. An ML application's quality is dominated by data and model behavior — which can degrade silently after deployment. So QA in ML means answering not only "does the code run?" but "is the data trustworthy?", "does the model behave well for all user groups?", and "does the deployed system survive production traffic?" The remainder of this lecture builds each of those answers, one layer at a time.

12.3 Traditional Software Testing vs Machine Learning Testing

Traditional software testing and ML testing differ fundamentally in what they verify, how they fail, and what drives the logic.

12.3.1 Logic Source

In traditional software, logic is explicitly programmed. Testing verifies that the code produces expected outputs. Data goes in, the program runs, and any errors are logged and fixed.

In machine learning, logic is learned from data. Testing must verify the learning process, data integrity, and statistical output. Everything starts from and runs around the data.

12.3.2 Failure Modes

Traditional software fails due to bugs — straightaway identifiable errors in the code.

ML fails due to concept drift, data corruption, or biased training data. These failures are subtler and harder to detect.

12.3.3 Shopping Example: Traditional vs ML Failure

Consider an e-commerce shopping scenario in a hot climate (such as Rajasthan in summer). A user browses summer wear — shirts, t-shirts, shorts — adds items to the cart, and checks out.

Traditional software failure: If there are billing errors or problems adding items to the cart, those are traditional software bugs. The checkout process, payment, and search functions are all code-driven.

ML failure: While shopping, the recommendation engine at the bottom of the page starts showing winter wear — overcoats, winter caps, socks, jackets — alongside summer items. The website itself works perfectly: no UI bugs, no billing issues, no search problems. But the recommendation model is giving wrong suggestions. This is a model training failure — the rules the model learned are not producing correct predictions.

The top layer (checkout, billing, search) is traditional software. The bottom layer (recommendations) is the ML model. When the model recommends winter wear during summer shopping, something is wrong with the ML testing, even though the software layer is fine.

12.4 Types of Software and ML Tests

Machine learning testing caters to three main dimensions (with a fourth, less common one).

12.4.1 The ML Testing Pyramid

Base — Data set tests: Everything around the data. Schema validations, unit tests for data cleansing, feature pipelines. The base includes all data-related testing.

Middle — Model tests: How the model performs — fairness, staleness, overfitting checks, robustness, scalability, and predictive performance. Everything around model behavior.

Top — Infrastructure tests (also called system tests): End-to-end pipeline testing, UI integration, latencies, and overall system behavior. Machine learning is only a small part of the entire system; infrastructure testing ensures the model integrates properly with APIs, databases, and other components.

Heuristic tests (less common): Common-sense rules that apply across all phases, most importantly on the data stage. For example, age cannot be negative — a basic sanity check that catches impossible values. Heuristic tests are not a main pillar but hold importance in each testing dimension.

12.4.2 Unit Testing in Software vs ML

Unit testing methods are conceptually the same — testing individual modules for correct functionality with varying inputs and checking outputs, including graceful error handling.

However, the way unit tests are performed differs. In traditional software, developers run code against data. In ML, unit tests verify the behavior of algorithms and models. These tests may relate to a module, a core business concept, or an entire application requirement. The scope differs from developer-centric testing to business-centric testing of algorithm behavior.

12.5 ML Ops Control Loops

The ML Ops control loop describes how testing and monitoring occur continuously throughout the ML lifecycle.

12.5.1 The Control Loop Flow

1. Raw data storage: Data arrives and gets validated.
2. Validation gate: If data fails validation, it gets quarantined and alerts are generated immediately. If it passes, it proceeds.
3. Feature stores: Data moves into different feature pipelines.
4. Training orchestrator: Governs the model training process.
5. Model registries and CI/CD: The model history and pipeline side. Metrics are generated and dashboards are populated.
6. Deployment to production: If everything passes — data, features, model — it deploys.
7. Monitoring: After deployment, monitoring continues for any drift. This does not stop at deployment.
8. Retraining loop: Monitoring triggers the next retraining cycle. Data flows into features, then training, then registry/serving, and finally monitoring — which triggers the next loop.

The key insight is that this is a continuous loop. The job does not finish after deploying to production. Monitoring keeps running, catching any drift, and the cycle repeats. The loop is "controlled" at every stage — metrics flow back into dashboards and trigger retraining when needed.

Data arrives → validation gate → feature stores → training → model registry + CI/CD
      ↑                                                                          ↓
   monitoring ←────────────────── deployment ←────────────────────────────────────
      ↑ (drift detected) → retraining loop repeats

12.6 Data Quality: The Foundation

The principle of "garbage in, garbage out" governs ML systems. Whatever data is fed to the model, the output will reflect that quality.

12.6.1 Why Data Quality Matters

Hospital temperature example: In a hospital, one ward measures patient temperature in Fahrenheit while another uses Celsius. At a surface level, both are measuring temperature. But the data quality differs because the units are inconsistent. No matter how much historical data is collected, predictions will always be wrong if the inputs are mixed. The model performance degrades and eventually the production system fails.

Data quality dictates the upper bound of model performance. Historical data may contain human biases that the model amplifies. Errors in data collection pipelines degrade production systems. Automated data validation is a strict requirement before any training begins.

12.6.2 Dimensions of Data Quality

Accuracy: Data reflects real-world truth. For example, correctly labeled customer churn status. Do not say "tomatoes are purple" — accuracy means matching reality.

Completeness: No missing records or null values. All user profiles have age and locations filled in. There is no ambiguity in the data.

Consistency: Data formats are uniform across the system. Date of birth should not be YYYY-MM-DD in one module and DD-MM-YYYY in another. Formats must be uniform.

Timeliness: Data is up to date and relevant. It should not come from ancient sources. Real-time stock prices, for example, must be delayed by less than one second to be useful.

12.7 Data Pipeline Validation

A data pipeline validation system checks records at gateway stages before they enter training. Each record passes through multiple validation gates.

12.7.1 Validation Gate Types

Schema check gate: Verifies missing columns or type changes (e.g., data-to-string conversions). This is the first gateway before data enters training.

Value check gate: Defines impossible ranges — negative ages, statistical outliers. This aligns with heuristic testing (common-sense rules).

Label health check gate: Checks for extreme class imbalances or missing target labels.

Action gate: If inputs do not match expectations, the pipeline automatically holds or quarantines the record.

12.7.2 Worked Example: Three-Record Validation

Three records are passed through the data validation pipeline:

Record 1: User ID with valid age (e.g., 25) and status "active." This record passes all validation gates — schema, value, and label checks all succeed.

Record 2: User ID with negative age (e.g., -5) and status "active." This record fails the value check gate. The error message: "Field age input should be greater than or equal to 0."

Record 3: User ID with the age column entirely missing and status "active." This record fails the schema check gate because the required age field is absent.

The output shows:

• Record 1 passes all gates and enters the pipeline.
• Record 2 fails with a validation gate error (negative age).
• Record 3 fails with a validation gate error (missing age column).
• The pipeline continues with one valid record out of three.

This shows how the validation pipeline catches errors at the gateway level, before data ever reaches the training stage. In real-world scenarios with lakhs or millions of records, these gates run at scale. The dimensions of accuracy, completeness, consistency, and timeliness must be well-defined and enforced continuously. The pipeline does not stop — it runs in iterative loops, checking every batch of data as it arrives.

12.8 Pipeline Quality

Pipeline quality ensures reproducibility and automation. No matter how many times a pipeline is run with the same input — days, weeks, or months later — it must produce the same exact outputs.

12.8.1 Key Principles

Reproducibility: Rerunning a pipeline with the same input must yield identical outputs. Data, code, and model artifacts must be explicitly linked and versioned together. If any component changes, the output changes.

Orchestration: Tools like Airflow and Kubeflow manage complex dependency graphs — if some code has a dependency on another part, the orchestrator manages it internally with well-maintained dependency tracking.

12.8.2 Continuous Integration for ML Pipelines

CI/CD in the ML context works as follows:

1. Code commits trigger automated syntax and unit tests.
2. Model architecture changes trigger training on small synthetic data sets (a form of unit testing for the model).
3. Data schema updates trigger validation checks against historical data sets.
4. Successful tests result in deployable artifacts that are ready for deployment to a central repository.

This is the check-in, check-out concept in CI/CD — all tests must pass before code enters the main pipeline.

12.9 Model Quality

Model quality goes beyond simple accuracy. Accuracy is often misleading, especially in highly imbalanced data sets.

12.9.1 Why Accuracy Alone Is Misleading

Face recognition example: A study showed that a US firm's face recognition model had less than 2–3% accuracy drop for fairer skin tones, but 10–15% accuracy drop for darker skin tones. When looking at the model overall, the accuracy appears very high — perhaps 95% or more. But when the data is broken into subgroups by skin type, the imbalance becomes visible. The model performs well on one demographic and poorly on another.

This is why model quality must be evaluated beyond aggregate accuracy. It includes:

• Interpretability: The model's outputs must be explainable — if a certain input is pushed, a likable and understandable output should result.
• Fairness: The data and predictions should not give absurd or biased outputs.
• Computational efficiency: The mathematical operations must be efficient.

Quality degrades over time and requires continuous validation. Evaluating quality requires testing on data strictly unseen during training.

12.9.2 Core Metrics: Precision and Recall

These are the two most important model quality metrics — extremely common in interviews, assignments, and vivas.

Precision measures accuracy: how accurate are the model's positive predictions?

\[ \text{Precision} = \frac{TP}{TP + FP} \]

where \(TP\) is true positive (correctly predicted positive) and \(FP\) is false positive (incorrectly predicted as positive — the model predicted positive but the answer was negative).

Recall measures completeness: does the model capture all actual positives?

\[ \text{Recall} = \frac{TP}{TP + FN} \]

where \(FN\) is false negative (the model predicted negative but the answer was positive — the model missed it).

F1 Score balances precision and recall for imbalanced data. It is the root mean square (harmonic mean) of precision and recall:

\[ F1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \]

If both precision and recall hold accurate values, the F1 score is also accurate. Otherwise, it produces large errors, especially in regression tasks.

12.9.3 Slicing and Subpopulation Testing (The Iceberg Model)

This is a critical evaluation technique. Think of an iceberg: what is visible above the water looks good, but the massive hidden portion below can cause disaster. The Titanic sank because it hit the iceberg below the waterline — everything looked fine above the surface.

Similarly, a model with 95% global accuracy might have only 50% accuracy on a minority demographic. The overall number hides the problem.

Slicing means dividing data sets into distinct subgroups — by location, gender, device type, skin type, etc. — and calculating performance metrics independently for each slice. This provides actionable insights into specific areas where the model requires mitigation or retraining.

12.9.4 Worked Example: Slicing Test with Desktop vs Mobile

Eight records are split into two subgroups: desktop (4 records) and mobile (4 records).

Actual labels: [1, 0, 1, 1, 1, 0, 1, 0] (first four = desktop, last four = mobile)

Model predictions:

• Desktop (first four): [1, 0, 1, 1] — all four match the actual labels perfectly.
• Mobile (last four): [0, 1, 0, 0] — only the fourth record matches.

Sliced accuracy:

• Desktop: 4/4 = 100% — all records predicted correctly.
• Mobile: 1/4 = 25% — only one record predicted correctly.

The global metric hides the fact that the mobile subgroup has critically poor performance. This is why slicing and subpopulation testing is essential — it reveals hidden failures that aggregate metrics mask.

12.10 Testing Model Training vs Model Inference

Understanding the difference between training testing and inference testing is essential. Combining them is analogous to combining functional and non-functional requirements in testing.

12.10.1 Training Testing (Functional)

Primary goal: Verify learning capability and convergence.

• Compute constraints: High throughput, long duration, batch processing.
• Failure modes: Exploding gradients, out-of-memory errors.
• Environment: Offline — isolated compute clusters.

12.10.2 Inference Testing (Non-Functional)

Primary goal: Verify prediction speed, format, and stability.

• Compute constraints: Low latencies, real-time response, high concurrency.
• Failure modes: API timeouts, incompatible input shapes.
• Environment: Online — production-facing, exposed via HTTP services and APIs.

The inference side relates to the previous session's demo where models were exposed as microservices via APIs. If those APIs timeout, that is an inference testing failure. Inference testing can be thought of as a pre-prod to prod scenario — testing how the model will behave when exposed to live users.

12.11 System Quality and Production Experimentation

12.11.1 System Quality: The Holistic View

An ML model is only a small percent — less than 5% — of the total codebase. It interfaces with the UI, databases, network stability, and more. A perfect model with a failing front-end results in zero business value. Clients and stakeholders think in terms of the entire system, not just the model.

System quality requires collaboration between data scientists, software engineers, testers, product managers, and SREs. The system must not compromise its quality at any level.

12.11.2 Observability in Production: Three Metric Types

Once deployed, the system requires continuous monitoring across three metric dimensions (plus alerting across all):

System metrics (infrastructure): CPU usage, memory consumption, network latencies.

Model metrics (production behavior): Confidence scores, anomaly rates, accuracies, precisions.

Business metrics: Click-through rates, conversion rates, revenue impacts.

Weather.com example: Suppose the model correctly predicts heavy rainfall in a region during monsoon season. But when users click on the interface to see details, they get a 500 HTTP error or the page hangs for 5–10 seconds. Even though the model metrics are good, the system and business metrics are failing — the user experience is ruined.

Alerting holds everywhere — automated thresholds must page engineers when metrics degrade significantly.

12.11.3 Why Offline Testing Is Not Enough

Offline data is static, but user behavior is dynamic and constantly changing. Model changes create feedback loops. Production experimentation provides the final, undeniable proof of business impact. Performance testing (applicable to certain applications) defines how the system performs under real conditions.

12.11.4 Traffic Testing Methodologies

Three specific methods for testing traffic in production:

A/B Testing (Split Traffic): Both the current model (version 1.0) and a new model (version 2.0) are exposed to live user traffic simultaneously. Users interact with both versions, and their return predictions are compared. Both models are in production and visible to end users.

Shadow Deployment: The current model serves 100% of user traffic. A shadow model receives copies of the same requests in the backend but is NOT exposed to users. Predictions are logged and compared internally. This is used when stakes are lower or when testing uncertain features. Shadow deployment is not recommended for banking models where features are still being discovered.

Canary Rollout: 99% of traffic goes to the current model. 1% goes to the new model (version 2.0). If the 1% shows positive results, the rollout gradually increases. "Canary" means rolling out bit by bit. This is granular and cautious.

The choice depends on business requirements:

• A/B testing is used most often across industries.
• Shadow deployment works for e-commerce and pharmacies but not recommended for banking (where stakes are high and features are still under discovery).
• Canary rollout is used when small, specific modules need testing with controlled risk.

12.12 Security for Machine Learning

ML models introduce unique vulnerabilities distinct from traditional web applications. Attackers can manipulate training data or inference requests. Security requires defending both intellectual property (model weights) and data privacy. Threat modeling must be integrated early in the ML lifecycle.

12.12.1 The Four Major Threats

Data poisoning (data collection phase): Injecting malicious records into the training data. Malicious actors push corrupted or deliberately misleading data during collection, corrupting the model's learned rules.

Backdoor attacks (model training phase): Manipulating algorithms by deliberately pushing certain inputs during training so the model learns incorrect patterns. The manipulators intensively push non-behavioral data to alter the model's precision and algorithm definitions.

Vulnerable libraries (deployment phase): Third-party and open-source libraries (Python packages, open LLM models) may contain vulnerabilities. When deploying, it is critical to verify that all dependencies are secure and do not create attack surfaces.

API serving attacks — adversarial input and model stealing: Malicious actors send unpredicted data to the model's API to discover trends and steal model behavior. They manipulate input data with adversarial effects to reverse-engineer and replicate the model.

Data collection and deployment phases are the most frequently targeted entry points for threat actors — data collection because anyone can inject malicious records, and deployment because open-source libraries are easy to manipulate.

12.13 RAGAS Framework for Evaluating RAG Systems

RAGAS (Retrieval Augmented Generation Assessment) is an automated framework for evaluating RAG pipelines. Traditional metrics like BLEU or ROUGE are inadequate for measuring hallucination and context relevance. RAGAS uses an LLM evaluator to score distinct components of a RAG pipeline without human intervention.

12.13.1 Two Pipeline Architectures

Offline indexing pipeline (data preparation):

1. Raw data arrives from sources (Wikipedia, project data sets, SharePoint, requirements documents).
2. Data is broken into chunks.
3. Chunks are embedded into numerical (vector) form via an embedding model.
4. Vectors are stored in a vector database.

Online retrieval pipeline (user query):

1. User submits a query.
2. The embedding model converts the query into vector form.
3. Vector search retrieves the top-K matching chunks from the vector database (e.g., top 4–5 matches).
4. The LLM generator (e.g., GPT-3.5, GPT-4.0) produces the final answer.

12.13.2 RAGAS Core Metrics

The metrics split into two categories — retrieval and generation:

Retrieval metrics:

• Context precision: Evaluates whether the top retrieved chunks are highly relevant to the query. This is about the precision of the vector search — are the results actually relevant?
• Context recall: Ensures the retrieval mechanism successfully fetched all necessary facts required to answer the question. This is about completeness — did the search miss anything?

High precision means no irrelevant data in results. High recall means no missing facts.

Generator metrics:

• Faithfulness: Measures whether the generated answer is strictly grounded in the retrieved context. Detects hallucination — does the LLM stick to what was retrieved, or does it fabricate?
• Answer relevance: Measures whether the generated answer directly addresses the user's initial query. Does it answer the actual question, or does it provide tangential information?

12.13.3 Precision vs Recall: Concrete Intuition

High precision example — spam filter: The spam model must be highly precise. High precision ensures important emails are not incorrectly sent to the spam folder. Every email marked as spam should actually be spam.

High recall example — airport bomb threat: Whether the threat is a hoax or genuine, the system must have high recall — it must catch every potential threat and trigger action. Missing a real threat (low recall) is catastrophic, even if it means investigating some false alarms.

12.13.4 Worked Example: RAGAS Evaluation

A RAGAS evaluation is run with hardcoded test data. The evaluator uses an LLM (based on an OpenAI API key) to score the pipeline.

Input questions:

• "What is the capital of France?"
• "What causes ocean tides?"

Evaluation results:

• Context precision: 1.0 (perfect — retrieved chunks are all relevant).
• Context recall: 1.0 (perfect — all necessary facts were retrieved).
• Faithfulness and answer relevance are also scored.
• Overall score: 75% (averaged across all metrics).

The framework displays results per question, showing faithfulness, relevance, precision, and recall for each query. This is useful for evaluating how accurately and completely a RAG pipeline performs.

Measurement tools: RAGAS uses foundation models (GPT-3.5, GPT-4.0, etc.) as automatic judges to score outputs.

12.14 Version Control and Containers (Introduction)

12.14.1 Git: Version Control

Git is a version control tool that tracks changes over time. Developers check out code from the main branch, work on feature branches, and merge back when done. Multiple developers can work in parallel on different feature branches.

Branching strategies:

• Never commit directly to the main branch.
• Use feature branches for active development.
• Use pull requests for peer reviews before merging.
• Never commit large data files or heavy model files to Git.

12.14.2 Dependency Management and Packaging

Packaging means bundling code with its dependencies. Python uses requirements.txt (pip-based lists of packages and versions) to manage external libraries. When packaging a Python project:

• Use proper directory structures.
• Include metadata (setup.py).
• Share via private PyPI servers or artifactories.
• Ensure all external dependencies (e.g., datasets, ragas) are included.

Dependency errors arise when packaged code does not include all required libraries.

12.14.3 Docker: Containers for ML

Docker is a container that bundles everything — code, external dependencies, drivers, OS — into a single image. This eliminates the "it works on my system" problem because the entire environment is packaged together.

A Docker workflow:

1. Start from a clean Python-based image.
2. Copy the dependencies file.
3. Run install commands.
4. Copy the code.
5. Define startup commands.
6. Push the container to the next environment.

Docker ensures reproducibility across development, testing, and production environments. Detailed hands-on work with Docker and Kubernetes will follow in subsequent sessions.

Exam Guidance Summary

Key Industry Applications

• E-commerce recommendations: The summer/winter wear example illustrates ML testing vs traditional software testing in a real shopping scenario.
• Face recognition bias: A US firm's model showed 10–15% accuracy drops for darker skin tones, demonstrating why aggregate accuracy is misleading and slicing tests are necessary.
• Weather.com: A model may predict correctly but the front-end may fail (HTTP 500 errors, latency), showing the importance of system-level metrics beyond model metrics.
• Banking sector: Shadow deployment is not recommended for banking models where features are still under discovery; stakes are too high.
• E-commerce and pharmacies: Shadow deployment is acceptable where stakes are lower.
• Hospital data quality: Mixing Fahrenheit and Celsius across wards degrades model predictions — a concrete example of consistency as a data quality dimension.
• Airflow and Kubeflow: Industry tools for orchestrating complex ML pipeline dependency graphs.
• RAGAS framework: Widely used in the market for evaluating RAG pipelines, using LLMs as automated judges (GPT-3.5, GPT-4.0).
• Docker containers: Industry standard for packaging ML environments to ensure reproducibility across dev, test, and production.
• Airports and security systems: High-recall systems must catch every potential threat regardless of whether it is a hoax — illustrating the recall concept in a real-world safety context.