Requirements Engineering and ML System Architecture
Prerequisite Knowledge
This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.
Previously Covered in This Subject
- The ML Pipeline — covered in Lectures 1-2 and Lecture 2 (stages: manage data, train model, evaluate, deploy, monitor)
- Deterministic vs Probabilistic Systems — covered in Lectures 1-2 and Lecture 2 (fundamental distinction between SE and ML)
- Software Engineering vs Machine Learning — covered in Lecture 2 (specification, requirements, summary of contrasts)
Requirements Engineering and ML System Architecture
3.1 ML and Non-ML Components in Real-World Systems
3.1.1 The Composition of ML-Enabled Systems
Hook: When you use a speech-to-text app, where does the "intelligence" actually live? Is it just the AI model, or is there more happening behind the scenes?
A production software system is rarely just a machine learning model — it is a collection of ML components and non-ML components working together. We saw this in the previous session when we identified that any system as a whole contains both types of components. The key insight is that the ML component, while central in many applications, is always surrounded by supporting infrastructure.
Intuition: Think of a speech recognition system like a hospital. The surgeon (ML model) performs the operation, but needs nurses, anesthesiologists, receptionists, billing staff, and administrators to deliver complete patient care. The surgeon is critical, but alone cannot run a hospital.
Consider a speech recognition or transcription system. The core ML component — the speech recognition module that takes audio input and produces real-time English transcription — is only one piece. A complete system also needs:
- User interface: Where users upload audio and view transcripts
- Authentication: User accounts, login, access control
- Storage: Audio/video upload, cloud storage for recordings
- Payment: Subscription or per-use billing
- Database: Persistent storage for transcripts and metadata
- Cloud processing: Backend infrastructure for scaling
- Logging and monitoring: System health and usage tracking
All of these work together to deliver the end-to-end service.
Worked Example: This Lecture Platform
The current lecture itself is delivered on a platform that embodies this architecture:
- Speaker's live video is uploaded to cloud storage after the session
- User accounts manage access (students, instructors)
- The institution handles licensing payments
- Speech recognition (transcription) converts spoken words to text
- A database stores recordings and transcripts
- Cloud processing handles the backend
- Logging and monitoring lets system administrators track live lectures
The block diagram of a transcription service applies directly to how this platform operates.
ML as Core vs. Supporting Component
The ML component can play different roles in the overall system:
| Role | Description | Example |
|---|---|---|
| Core/Heart | Primary value proposition of the system | Speech recognition in transcription service |
| Supporting/Auxiliary | Enhances product without being main feature | Audit risk prediction in tax software |
A tax software application illustrates the supporting pattern. The heart of tax software is tax computation — calculating liabilities, deductions, and returns according to tax law. But alongside that core, you can add an audit risk prediction module that uses ML to assess whether a given tax filing is likely to be flagged for audit. Here, the ML is an add-on functionality that enhances the product without being the primary feature.
The ML Pipeline Behind Every Component
When we say "the ML component" in a production system, we mean a specific piece of functionality — speech recognition, audit risk prediction, object detection — that is supported by its own machine learning pipeline behind the scenes. That pipeline includes:
- Model requirements gathering
- Data collection
- Data cleansing
- Labeling
- Feature engineering
- Model training
- Deployment
- Ongoing monitoring
This lifecycle exists for every ML model in production, whether it is a predictive AI model, a generative AI model, or an agentic AI model.
Pitfall: Assuming the ML model IS the product. In reality, the model is one component surrounded by substantial infrastructure. A model without deployment, monitoring, and integration is just a notebook experiment.
Recap: Production ML systems combine ML models with traditional software infrastructure. The model can be the core value or a supporting feature, but either way requires a complete pipeline from data to deployment.
3.1.2 Student Questions on ML-System Integration
Q: In any real-time scenario, is there a single end-to-end pipeline for both ML and non-ML components, or separate pipelines?
A: They are different pipelines. The traditional software engineering process follows established methodologies — Agile, Scrum, Extreme Programming — where you have user requirements, implement them in sprints of two to four weeks, do unit and integration testing within each sprint, and deliver to the customer. The ML process is fundamentally different because it is exploratory in nature. You cannot plan ML work in the same deterministic way. Exploration — particularly Exploratory Data Analysis — must happen before you can even define model requirements. The two pipelines run in parallel and integrate at specific points. Later sessions will walk through a full end-to-end education technology system showing exactly how the SE pipeline (with its own CI/CD) and the ML pipeline (with its own process) come together to form one application.
Q: Does data become part of the software system logic itself?
A: Let me ground this with a concrete example from healthcare. Consider a COVID data prediction project with real patient data — 1,200 patient records collected during the pandemic. The goal was to build a classifier that, when a new patient is admitted, predicts whether they will recover or expire. Multiple classification models were trained — SVM, Random Forest, Decision Tree — and Random Forest gave the best accuracy. But developing the model is only half the battle. The model must be integrated into the existing hospital management software. The model is deployed as an API endpoint — typically an HTTP POST endpoint. When a nurse enters a new patient's data (age, demographics, specific clinical parameters) into the hospital system, that system calls the model's endpoint, gets a prediction back, and acts on it. If the prediction says "will recover," no additional intervention is needed. If it says "will expire," additional interventions are triggered. So the non-ML hospital system leverages the ML model through an API — the model is developed and deployed separately.
Real-world Pattern: This API-based integration pattern — model as a service behind an HTTP endpoint consumed by a larger application — is the dominant deployment pattern in production ML systems today.
3.2 Software Engineering Process vs Machine Learning Process
3.2.1 Experiment-Driven Data Development
Hook: Why can't we just use Agile sprints for ML projects? What makes ML development fundamentally different from traditional software?
The two processes differ fundamentally in their nature and must be understood as separate but coordinating disciplines.
Intuition: Think of traditional software engineering like building a house from a blueprint — you know what you're building, follow the plan, and test against specifications. ML development is like scientific research — you form hypotheses, experiment, and the path forward depends on what you discover. You cannot predict upfront what algorithm will work or what patterns exist in the data.
The Software Engineering Process
The SE process follows a structured lifecycle:
- Requirements engineering → Capture what the system should do
- Specification and design → Architecture decisions
- Implementation → Write code
- Testing → Verify against requirements
- Deployment → Release to production
- Operations → Monitor and maintain
When you reach the design phase, you must consider all components — both software and ML — which is where the ML process gets triggered.
The Machine Learning Process: Experiment Driven Data Development
The ML process is fundamentally exploratory:
- Determine whether ML is suitable for this application
- Data exploration (EDA) → Understand what data exists
- Capture model requirements → What should the model predict?
- Model development → Training, evaluation, selection
- Model integration → Plug into the SE pipeline
- System testing → Combined SE + ML testing
- Operationalize → Monitor the ML model continuously (24×7)
Key Difference: These two processes run in parallel, not sequentially. They integrate at a specific integration point where the trained model is plugged into the larger application. Trying to force ML into traditional SDLC phases fails because ML is inherently exploratory.
3.2.2 Operations, Data Drift, and Model Versioning
Q: Why are there separate operations boxes for both SE and ML in the diagram? Shouldn't integration just go through system testing?
A: Operations for the SE application and operations for the ML model are fundamentally different disciplines — this is why MLOps, LLMOps, and AgentOps are becoming separate fields.
| Aspect | SE Operations | ML Operations |
|---|---|---|
| Infrastructure | Kubernetes (EKS, ECS) | Same Kubernetes cluster |
| Metrics | CPU usage, memory, application health | Data drift, concept drift, prediction accuracy |
| Concerns | Worker nodes, microservices health | Model staleness, accuracy degradation |
| Response | Restart services, scale resources | Retrain model, update data pipeline |
Same infrastructure, different monitoring concerns, so separate operational tracks.
Q: When data keeps changing — as with COVID data that evolved rapidly — the model must be retrained. Does the entire SDLC get affected?
A: This is exactly why the SE process is generally separated from the ML process. We have the concepts of data drift and concept drift:
- Data drift: Statistical properties of input data change over time
- concept drift: The relationship between inputs and outputs changes
When the statistical properties of the input data change, the model must be retrained and redeployed. But from the SE system's perspective, what it receives is always the same thing: an API endpoint that accepts a request and returns a prediction. Whether the model behind that endpoint is version 1.1 or 1.2 may not matter to the consuming application.
Worked Example: ChatGPT Model Evolution
Think of ChatGPT: we have gone from GPT-3.5 to 4, 4.5, 5.1, 5.2, 5.3, and now 5.5. Each is essentially a new model. Applications using GPT behind the scenes decide whether to upgrade based on the new model's capabilities — better NLP, better vision, better sentiment analysis.
The ML side keeps evolving with new models and data versions; the SE side decides which version to invoke, based on accuracy, cost, and capability trade-offs. Data versioning and model versioning are handled independently within the ML pipeline.
Q: If incrementally changing training data causes model predictions to flip (e.g., a patient previously classified as "will recover" now classified as "will expire"), how do we handle that?
A: This comes down to bias and variance in the model. If the dataset is imbalanced — not containing all possible classifier outcomes proportionally — you get high bias or high variance, leading to unreliable predictions. This is why data cleansing and pre-processing are critically important. If the dataset has high variance or high bias, the model will be incorrect regardless of the algorithm. We are constantly collecting real data, feeding it into the model, and checking whether the same algorithm still works or whether a new algorithm is needed. But throughout this cycle, pre-processing quality and the EDA process remain paramount — garbage data means garbage predictions.
Q: What is the difference between "integrate ML component" and "operate ML component"?
A:
| Activity | Nature | Duration | Description |
|---|---|---|---|
| Integration | One-time | Finite | Wire the consuming application to call the model's HTTP POST endpoint with requests and receive responses |
| Operation | Continuous | 24×7, 365 days/year | Continuously monitor the model in production using ML-specific metrics (different for MLOps, LLMOps, AgentOps) |
Integration happens once; operation never stops.
Q: Why is there a connection between the two operations boxes?
A: Because the systems are in coordination with each other, not independent. The model could be down while the hospital management system is up, or the consuming system could be down while the model is healthy. They depend on each other — the application needs the model to deliver its functionality, and the model's value is realized only through the application.
Pitfall: Assuming that once a model is deployed, the job is done. In reality, deployment is just the beginning. Models degrade over time due to data drift and concept drift, requiring continuous monitoring and periodic retraining.
Recap: SE and ML are separate but coordinating disciplines. SE follows structured methodologies; ML is exploratory. They run in parallel and integrate at defined points. Operations are fundamentally different — SE monitors system health, ML monitors model health. Integration is one-time; operation is continuous.
3.3 ML Pipeline and Lifecycle
3.3.1 The Standard ML Pipeline Phases
Hook: What happens behind the scenes when you ask Siri a question? What does the journey from raw audio to intelligent response look like?
Behind every ML component in a production system lies a standard machine learning pipeline. This pipeline is most fully defined for predictive AI but generalizes across generative and agentic AI as well.
The 10 Phases of the ML Pipeline
| Phase | Description | Key Activities |
|---|---|---|
| 1. Model requirements | What should the model do? | Define accuracy targets, latency constraints, business goals |
| 2. Data collection | Gathering raw data | Sources, volume, quality checks |
| 3. Data cleansing | Handling data quality issues | Missing values, outliers, inconsistencies |
| 4. Data labeling | Assigning ground truth | For supervised learning; can be manual or automated |
| 5. Feature engineering | Creating input features | Select, transform, and create predictive variables |
| 6. Model training | Applying algorithms | Train multiple models, hyperparameter tuning |
| 7. Model evaluation | Measuring performance | Held-out data, cross-validation, metrics |
| 8. Deployment | Making model available | API endpoint, containerization, scaling |
| 9. Model monitoring | Observing behavior | Performance tracking, drift detection |
| 10. Retraining | Updating the model | When data drifts or performance degrades |
Critical Insight: The model lifecycle is not linear. Retraining loops back to data collection and feature engineering. This creates a continuous cycle, not a one-time process.
Versioning: Tracking the ML Artifact Lineage
Two types of versioning are essential:
- Data versioning: Tracks which dataset was used to train which model version
- Model versioning: Tracks which model artifact is currently serving predictions
This lineage is critical for reproducibility, debugging, and rollback capabilities.
Visual: The ML Lifecycle Cycle
Model Requirements → Data Collection → Data Cleansing → Data Labeling
↓
Retraining ← Model Monitoring ← Deployment ← Model Evaluation ← Model Training
↓ ↑
└──────────────────── Data Collection / Feature Engineering ─────────┘The cycle continues as long as the model is in production.
Pitfall: Treating the ML pipeline as a one-time process. In reality, production ML systems require continuous monitoring and periodic retraining as data evolves.
Recap: Every ML component has a standard pipeline from requirements through deployment. The lifecycle is cyclical, not linear — retraining loops back to earlier phases. Data and model versioning are essential for tracking lineage and enabling rollback.
3.4 Roles in ML-Enabled Systems
3.4.1 Data Scientist, ML Engineer, and Software Engineer
Hook: Who does what in an ML project? Is it one person doing everything, or are there specialized roles?
Different phases of the ML lifecycle engage different roles, and the boundaries between them matter for system design.
The Three Key Roles
| Role | Primary Focus | Key Skills | ML Lifecycle Phases |
|---|---|---|---|
| Data Scientist | Building the best model | Data analysis, algorithms, statistics | Requirements, EDA, model development |
| ML Engineer | Deploying and operating models | DevOps, infrastructure, monitoring | Integration, deployment, operations |
| Software Engineer | Building non-ML components | Frontend, APIs, databases, microservices | SE application development |
Worked Example: Role Responsibilities
Data Scientist:
- Determines whether ML is suitable for the application
- Performs Exploratory Data Analysis (EDA)
- Develops and evaluates models
- Works primarily with data and algorithms
- May not need deep SE knowledge
ML Engineer:
- Takes model from data scientist
- Deploys to production infrastructure
- Ensures API contracts are maintained
- Sets up monitoring (data drift, concept drift)
- Manages CI/CD pipeline for model updates
- Bridges data science output with SE system
Software Engineer:
- Builds non-ML components (React frontend, PostgreSQL database, microservices)
- Integrates with deployed ML models via API endpoints
- Core domain is the traditional software stack
Real-world Pattern: A full-stack developer building a React frontend with a PostgreSQL backend and multiple microservices is doing core software engineering. When one of those microservices needs a prediction, it calls the ML model's endpoint, gets the result, and uses it within the application flow.
Pitfall: Assuming one person can handle all roles effectively. In practice, the skills for data science (statistics, algorithms) are different from ML engineering (DevOps, infrastructure) and software engineering (frontend, APIs, databases).
Recap: Three distinct roles support ML systems: Data Scientist (builds models), ML Engineer (deploys and operates), Software Engineer (builds non-ML components). Clear boundaries enable effective collaboration.
3.5 Case Study: Apollo Autonomous Driving
Hook: How does a self-driving car see the world? What does it take to build a system that can navigate complex roads without human intervention?
The Apollo autonomous driving platform illustrates how ML and non-ML components integrate in a safety-critical, real-time system. This is a case of Advanced Driver Assistance Systems (ADAS), which are categorized from Level 0 (fully manual) to Level 6 (fully autonomous). Most production cars today are at Level 2 (cruise control), while fully autonomous operation at Levels 5–6 remains largely experimental, especially in complex road environments.
3.5.1 Hardware and Sensor Suite
Three Categories of Hardware Sensors
| Sensor | Function | Strengths | Limitations |
|---|---|---|---|
| Camera | Visual images of road | Lane detection, traffic lights, visual obstacles | Affected by lighting, weather |
| Radar | Radio-based detection | Distance, speed measurement; works in poor visibility | Lower resolution |
| LIDAR | Light-based detection | Precise 3D map, 360° view, object classification | Expensive, affected by weather |
The car must simultaneously:
- Know its lane
- Detect traffic signals (red/green/yellow)
- Maintain awareness of every object (front, right, left, rear) and their speeds
3.5.2 The 28 ML Models
Worked Example: Apollo's Multi-Model Architecture
Apollo uses approximately 28 different ML models working in concert:
Traffic Light Perception Pipeline:
Camera → Pre-processing → Traffic Light Detection → Post-processing → Color Classification → PredictorLane Detection Pipeline:
Camera → Pre-processing → Lane Detection Model → Post-processing → Lane Result → PredictorCamera Obstacle Detection:
- Camera-based object detection for obstacles in the vehicle's path
LIDAR Obstacle Detection:
3D Point Cloud → Pre-processing → Obstacle Classification (bicycle, pedestrian, vehicle, animal) → PredictorKey Insight: Multiple models run in parallel and their outputs feed into a central predictor that determines the vehicle's next action. Model types include YOLO 3D, CNN-based models, and RNN-based models.
3.5.3 Key Architectural Insights
Critical Insight: ML System ≠ Standalone Model
A real-world system uses multiple models that collectively deliver the ML functionality. Single-model thinking does not scale to production systems.
Five Architectural Insights
- Large-scale ML architecture: 28 models handle different perception tasks, working in unison for autonomous driving
- Model interaction is complex: Outputs can feed into other models or non-ML components.
Interactions can be:
- Sequential (pipeline)
- Parallel (simultaneous inference)
- Hybrid
- Code plays a critical role: Pre-processing and post-processing logic (edge detection, image normalization, data format transformation) sits between raw sensor data and ML models
- Multi-source data fusion: Cameras (images), LIDAR (3D point clouds), radar (distance/velocity) — different data modalities must be normalized and fused into a single driving decision
- SE-for-ML challenge: Designing a system where ML and non-ML components come together in a highly complex, real-time, safety-critical setting
3.5.4 Handling Failures in Autonomous Systems
Q: How does an autonomous vehicle handle erratic pedestrian behavior, stray animals, sudden potholes, or the dilemma of braking suddenly when a vehicle is close behind?
A: The ADAS levels (0–6) define how autonomous the system is and what failure modes are acceptable:
| Level | Description | Failure Tolerance |
|---|---|---|
| Level 0 | Fully manual | Human handles all failures |
| Level 2 | Cruise control | System assists, human monitors |
| Level 3–4 | Conditional automation | Human expected to intervene in non-ideal situations |
| Level 5–6 | Full autonomy | No human intervention; may not be applicable to complex environments |
For critical components like object detection, a failure means a potential crash — so continuous monitoring is essential. The system may use:
- Redundancy: Multiple models for the same task
- Fallback: Switching to a backup model or deployment region
- Graceful degradation: Operating at reduced capability rather than failing entirely
Q: Do these models run embedded on the car or communicate with a remote server?
A: They run on embedded hardware — automotive ECUs (Electronic Control Units) such as NVIDIA Drive AGX or Tesla's FSD Computer. These are real-time embedded systems designed to run ML inference locally on the vehicle without requiring network connectivity, which is essential for safety in remote areas.
Q: What pre-processing and post-processing happens while the car is moving?
A: Raw sensor data — camera images, LIDAR point clouds — is rarely in the exact format that ML models expect.
Pre-processing examples:
- Edge detection
- Texture analysis
- Image normalization
- Coordinate transformation
Post-processing:
- Converts model's raw output (e.g., bounding box coordinates) into actionable information for the car's control system
Q: When the output of one model feeds into another and the final prediction is wrong, how do we trace the failure?
A: This is a fundamental challenge in composed ML systems. In a real-time environment, one component's failure can cascade. The approach is to identify critical versus non-critical components:
| Component Type | Failure Impact | Examples |
|---|---|---|
| Critical | Cannot fail without causing a crash | Object detection, traffic light recognition |
| Non-critical | Can degrade without catastrophic consequences | Infotainment, comfort features |
The ADAS level determines what can and cannot fail — a Level 6 system cannot fail at all, while a Level 2 system tolerates more component failures. Continuous monitoring of each model's output accuracy is essential for critical components.
Real-world: The Apollo case study is available as a research paper with detailed architecture descriptions and a companion YouTube video explaining LIDAR, camera, and radar integration for non-automotive readers.
3.5.5 Microsoft Case Study (Self-Study)
Self-Study Resource: Microsoft published a case study (2021–2022) documenting how they use AI/ML-based systems internally. They defined a 9-stage ML workflow used across their products — given that Microsoft has numerous products using AI/ML (Azure services, Office, etc.). The paper — titled "Software Engineering for Machine Learning" — provides best practices for adopting software engineering practices in ML development.
Pitfall: Thinking of ML systems as single models. Production systems like Apollo use 28+ models working together. The architectural challenge is managing model interactions, data fusion, and failure handling.
Recap: Apollo demonstrates production ML at scale — 28 models, three sensor types, real-time processing on embedded hardware. The key insights: ML systems are multi-model ensembles, code (pre/post-processing) is critical, and failure handling requires redundancy and graceful degradation.
3.6 When to Use Machine Learning
Hook: Should every software problem use AI? When is ML the right tool, and when is it overkill?
Not every problem requires machine learning. A fundamental question for any project is: Is ML the right approach? Three conditions strongly indicate that ML is appropriate.
3.6.1 Intrinsically Hard Problems
Definition: Intrinsically hard problems resist rule-based solutions because the underlying domain is simply too complex to encode manually.
Worked Example: Why Language is Intrinsically Hard
Language is the canonical intrinsically hard problem. The history of chatbots illustrates why:
Timeline:
- 1956: First chatbot built
- 70 years of rule-based chatbot development
- Modern LLMs (GPT, Gemini) achieve near-perfect understanding
Why rule-based chatbots failed:
- Had to parse prompts, identify intent, map to responses
- Required encoding ALL grammar rules and vocabulary
- Nearly impossible as number of intents grew
Inherent properties of language that rules cannot handle:
| Property | Challenge | Example |
|---|---|---|
| Ambiguity | Multiple valid interpretations | "I saw the man with the telescope" |
| Sarcasm | Literal meaning contradicts intent | "Oh great, another meeting" |
| Context | Words derive meaning from surrounding text | "bank" (river vs. financial) |
| Cultural nuances | Varieties differ in idiom and semantics | US vs. UK vs. Indian English |
Solution: Machine learning — particularly transfer learning — learns these patterns from massive text corpora rather than requiring explicit encoding.
Vision: The Second Intrinsically Hard Problem
Object detection in images varies dramatically with environment:
- A street in Bangalore looks nothing like the Antarctic
- You cannot write rules to cover every possible object in every possible setting
Solution: ML models learn visual features from data, making them robust to environmental variation.
3.6.2 Big Data Problems
Worked Example: One Minute of Internet Activity
| Platform | Volume per Minute |
|---|---|
| Google searches | 6.3 million |
| WhatsApp messages | 41.6 million |
| Instagram Reels | 694,000 |
| ChatGPT prompts | ~7,000 |
This is the big data deluge — data generated continuously at a scale that makes manual analysis impossible.
Why ML is Essential for Big Data
When you need to:
- Recommend songs to millions of users from tens of millions of tracks
- Generate personalized YouTube and Netflix recommendations
- Process millions of transactions per second
Rule-based logic cannot scale. The system must operate automatically, learning patterns from user behavior as data streams in. Bulk personalization at web scale is fundamentally an ML problem.
3.6.3 Time-Changing Problems
Worked Example: Fraud Detection
The problem: A rule like "flag any transaction above ₹50,000 made at night" may work for a while, but fraudsters adapt.
Why rules fail:
- Fraud patterns evolve continuously
- Variations by geography (cosmopolitan city vs. village)
- Variations by customer type (retail vs. commercial)
- Variations by demographics and transaction history
Solution: ML models can retrain on new data as patterns shift, making them suitable for problems where the underlying distribution changes over time.
Q: Are there additional categories beyond these three?
A: Student-contributed examples from the discussion:
- Deterministic vs. probabilistic reasoning
- Recommendation systems
- Medical coding of patient documents
- Supply chain management
- Loan EMI prediction
- Automatic customer service calls
- Space mission planning in unknown territories
- Claim authorization
Most of these fall under time-changing or big data categories.
Key Principle: If a problem is simple enough that a rule-based system can solve it (e.g., 2 + 2 = 4), you do not need ML. ML is for problems where data volume, complexity, or rate of change makes rule-based solutions impractical.
Recap: Three triggers for ML: (1) Intrinsically hard problems (language, vision), (2) Big data problems (volume exceeds human processing), (3) Time-changing problems (patterns evolve). If rules work, use rules. ML is for when complexity, volume, or change rate makes rules impractical.
3.7 Requirements Engineering for ML Systems
3.7.1 Deterministic vs. Probabilistic Systems
Hook: Can you guarantee that an ML system will always give the same answer to the same question? Why or why not?
Deterministic vs. Probabilistic Systems
| Aspect | Traditional Software | ML Systems |
|---|---|---|
| Behavior | Same input → same output always | Same input → may produce different outputs |
| Example | Calculator: 2 + 2 = 4 | ChatGPT: same prompt may get different responses |
| Reason | Fixed logic, no learning | Depends on training data, model version, prompt phrasing |
Traditional software systems are deterministic: the same input always produces the same output. If you ask a calculator for 2 + 2, it always returns 4. ML systems are probabilistic: the output depends on the data the model was trained on and the patterns it learned. The same input given today and three months later may produce different outputs because:
- The model has learned from additional data in the interim
- A new model version was deployed
- The prompt was phrased slightly differently
Three Concerns for ML Requirements Engineering
This probabilistic nature means requirements engineering for ML must account for:
- Data quality and availability: Does the required data exist? Is it labeled? Is it representative?
- Model behavior under uncertainty: What is acceptable performance? What is the error tolerance?
- Continuous learning and updates: How often will the model be retrained? What triggers retraining?
ML Requirements Engineering Scope
Requirements engineering for ML systems is about translating a high-level goal into measurable, data-aware, and model-aware specifications.
Traditional SE requirements:
- Talk to the business
- Capture requirements in use cases
- Follow established method (Scrum, Agile)
ML requirements must additionally address:
- The data component
- The algorithm/model component
- The deployment component
- All while accommodating probabilistic behavior
3.7.2 Industry Experiences with ML Requirements
Q: From industry experience, how are requirements for ML applications currently captured?
A: Multiple students shared their experiences:
- Oil and gas: Energy flow prediction across different energy sources (hydrocarbon vs. wind farm) with similar platform interfaces but different underlying models
- Storage industry (NetApp): Using unstructured customer support data in a data lake with ETL workloads to predict maintenance needs and suggest remediation steps
- Ad tech: Using product requirement document templates that scope use cases, identify required data inputs, and determine ETL needs before building audience segmentation and reach prediction models
- Anomaly detection: Transitioning from traditional SE requirements to ML revealed that the end goal was unclear — the team knew what data they had but not what patterns would emerge or what was achievable. This motivated the need for a specialized ML requirements framework
- Medical equipment (Medtronic): Using Voice of Customer and Voice of Business with ML embeddings to rank requirements by confidence scores; building an FMEA engine that generates failure mode predictions from user environments and system architecture
- Healthcare: Government-mandated authorization processing where ML makes approval recommendations from provider documentation within tight timelines
Key Insight from Industry: Transitioning from traditional SE requirements to ML revealed that the end goal was often unclear — teams knew what data they had but not what patterns would emerge or what was achievable. This motivated the need for specialized ML requirements frameworks.
Recap: Traditional software is deterministic; ML is probabilistic. Requirements engineering for ML must address data quality, model behavior under uncertainty, and continuous learning. Industry experiences show the need for specialized ML requirements frameworks.
3.8 Goals in ML-Enabled Systems
Hook: How do you know if your ML system is successful? What does "success" even mean for a system that makes predictions?
Every system starts with a goal. For ML-enabled systems, goals exist at four levels.
3.8.1 The Four Goal Levels
The Goal Hierarchy
| Level | Description | Example (Bank) | Example (University) |
|---|---|---|---|
| Organizational | Highest-level objective | "Reduce loan default rate by 15%" | "Reduce student dropout rate by 20%" |
| User | What specific roles need to achieve | Loan agent: process applications efficiently | Mentor: identify at-risk students |
| System (Product) | What the system must accomplish | "Process loan applications end-to-end" | "Track engagement, flag at-risk students" |
| Model | Specific ML prediction task | "Predict credit risk for applicant" | "Predict dropout probability" |
Key insight: Not every system goal requires a model goal. If the application is purely deterministic (e.g., a full-stack CRUD application with no prediction needs), there may be no model goal at all. A model goal exists only when one of the three ML triggers applies — the problem is intrinsically hard, involves big data, or is time-changing.
3.8.2 Goal Relationships: Supporting vs. Conflicting
Worked Example: Supporting Relationship
Model accuracy improves user experience.
When Gemini's first model launched in 2022–2023 with poor accuracy, user adoption was low despite Google's brand. As accuracy improved with Gemini 3.3, user experience and adoption followed.
This is a positive correlation — improving one goal directly supports another.
Conflicting Relationships: Trade-offs to Manage
| Trade-off | Description | Example |
|---|---|---|
| Accuracy vs. Latency | More accurate model may take longer | 2 seconds/90% accuracy vs. 30 seconds/95% accuracy |
| Cost vs. Quality | Better models cost more | GPT-5.5 (excellent, expensive) vs. GPT-Nano (cheaper, lower quality) |
Example scenarios:
- Autonomous driving: 30 seconds latency is fatal — must prioritize speed
- Batch document summarization: Extra accuracy may be worth the wait
This trade-off is universal in ML system design: better models cost more to run, and the decision of which model tier to use must balance budget against quality requirements.
Key Insight: Goal-setting for ML systems is not just about listing objectives — it is about acknowledging and managing conflicts. An organizational goal of "improve customer experience" may conflict with a model goal of "use the most accurate model" when that model's latency degrades the experience.
Recap: ML systems have four goal levels: organizational, user, system, and model. Goals can support each other (accuracy → user experience) or conflict (accuracy vs. latency, cost vs. quality). Effective goal-setting requires acknowledging and managing these trade-offs.
3.9 GR4ML: Goal-Oriented Requirements for Machine Learning
3.9.1 Why a Modeling Notation?
Hook: How do you write requirements for a system that learns from data? Can you use the same notations as traditional software?
Language is ambiguous. Two people can read the same requirement statement and interpret it differently. A modeling notation solves this by assigning fixed meanings to visual symbols — anyone trained in the notation interprets a given diagram identically.
UML vs. GR4ML
| Aspect | UML | GR4ML |
|---|---|---|
| Purpose | Traditional software modeling | ML-specific requirements |
| Strengths | Use cases, class diagrams, sequence diagrams | Data characteristics, algorithm choices, deployment |
| Limitations | Cannot capture ML-specific dimensions | Young framework (2021), growing adoption |
| Notation | Ovals, rectangles, arrows | Strategic goals, decision goals, actors, questions, indicators, insights |
ML systems introduce additional dimensions that UML was not designed to address. ML requirements must specify data characteristics, algorithm choices, deployment constraints, and probabilistic behavior. GR4ML — Goal-Oriented Requirements for Machine Learning — is a conceptual modeling framework proposed by a leading research institution (2021) designed specifically to capture requirements for ML applications, whether predictive, generative, or agentic.
3.9.2 The Three Views
GR4ML's Three Complementary Views
| View | Focus | Questions Answered |
|---|---|---|
| Business View | What and Why | Who are stakeholders? What are goals? How measure success? |
| Data Preparation View | Data | What data exists? What format? How does it flow? |
| Analytics Design View | Model | Which algorithms? What metrics? How deploy? |
These views are analogous to how UML provides different diagram types (use case, class, sequence, component, deployment) for different concerns.
3.9.3 Business View Components
Six Notation Elements of the Business View
| Element | Description | Notation |
|---|---|---|
| Strategic Goal | High-level business objective | Elliptical (oval) shape |
| Decision Goal | Specific lower-level goal supporting strategic goal | Elliptical with "D" annotation |
| Actor | Stakeholder role desiring the goal | Stick figure (like UML) |
| Question | Question that must be answered | Elliptical with "Q" annotation |
| Indicator | Measurable metric quantifying success | Specific graphical notation |
| Insight | What data actually reveals vs. what indicator targets | Data-driven reality check |
Key insight: Goals without measurable indicators are not actionable. An indicator might say "target 8 CGPA," but if historical data shows a student has consistently scored 4 CGPA across five semesters, the insight says the target is unrealistic.
3.9.4 Example: University Student Dropout Prediction
Worked Example: Complete GR4ML Business View
Strategic Goal: Reduce the student dropout rate by 20%.
Actors: Dean, Director, program heads — anyone with a stake in student retention.
Decision Goals:
- Is student intervention required? — Should we actively reach out to at-risk students?
- Should mentors contact specific students? — Which specific students should mentors reach out to?
Question: Which students are likely to drop out? Which students show low engagement?
Insights (data variables feeding the prediction):
- Daily student attendance records
- Assignment submission history
- Class participation metrics
- (N additional factors as available)
Model prediction: "Student A has an 85% probability of dropping out."
Indicator: Reduce dropout rate by 20% in the next academic year.
The complete chain:
Strategic Goal → Decision Goals → Question → Data Insights → Model Prediction → IndicatorThis forms a complete business view of the ML-enabled system at the requirements level, before any architecture or design work begins.
3.9.5 Example: Credit Risk Prediction in a Bank
Worked Example: GR4ML for Credit Risk
Actor: Loan processing agent at a bank.
Strategic Goal (implied): Minimize loan defaults while maximizing approved loans.
Decision Goal: Process a credit application — decide whether to approve or reject a given applicant.
Question: What is the credit risk of the current applicant?
Model Details:
| Parameter | Value |
|---|---|
| Input | Applicant profile — demographic data (age, city), qualification, financial history |
| Output | Credit risk classification — high risk or low risk |
| Model type | Binary classifier (predictive AI) |
| Usage frequency | Per instance — every applicant triggers a prediction |
| Update frequency | Monthly — retrain with new data to account for data drift |
| Learning period | 48 months (4 years) — trained on historical loan data |
Decision flow:
- If low risk → loan processes normally
- If high risk → application flagged for additional review or rejection
The model is retrained monthly on four years of rolling historical data, ensuring it stays current with changing economic conditions and fraud patterns.
3.9.6 GR4ML Adoption and Context
GR4ML Adoption Timeline
| Framework | Proposed | Became Standard | Time to Adoption |
|---|---|---|---|
| UML | 1990s | Early 2000s | ~10 years |
| Enterprise Ontology Model | 2006 | 2020s | ~20 years |
| GR4ML | 2021 | Growing adoption | TBD |
GR4ML is relatively young as modeling notations go. Conceptual models typically take years to gain traction and become standards. GR4ML's adoption is growing, particularly because it addresses a genuine gap — UML's inadequacy for ML requirements — and because it comes from a top-tier research institution.
Real-world: Research papers on GR4ML are available with detailed use cases. Search for "GR4ML" to find the original publications and case studies from the researchers who proposed it.
Pitfall: Using UML for ML requirements. UML cannot capture data characteristics, algorithm choices, deployment constraints, and probabilistic behavior. Use GR4ML or similar ML-specific frameworks.
Recap: GR4ML is a goal-oriented requirements framework for ML with three views: Business (what/why), Data Preparation (data), Analytics Design (model). The Business View uses six elements: strategic goal, decision goal, actor, question, indicator, insight. It addresses UML's inadequacy for ML requirements.
3.10 Alternative Methodologies
3.10.1 CRISP-DM, SEMMA, and Big Data Analytics Lifecycle
Hook: Is there a standard process for ML projects, like Agile for software? What frameworks guide ML development end-to-end?
While GR4ML focuses specifically on requirements capture, several end-to-end methods exist for the broader ML lifecycle.
Three Major ML Methodologies
| Methodology | Origin | Focus | Phases |
|---|---|---|---|
| CRISP-DM | Data mining industry | End-to-end ML lifecycle | Business understanding → Data understanding → Data preparation → Modeling → Evaluation → Deployment |
| SEMMA | SAS | Modeling phase | Sample → Explore → Modify → Model → Assess |
| Big Data Analytics Lifecycle | Modern big data | Full lifecycle | Discovery → Data preparation → Model planning → Model building → Communication → Operationalize |
CRISP-DM: The Industry Standard
CRISP-DM (Cross-Industry Standard Process for Data Mining) is the most widely adopted:
- Business understanding: What problem are we solving?
- Data understanding: What data do we have?
- Data preparation: Clean, transform, feature engineer
- Modeling: Train and evaluate models
- Evaluation: Does the model meet business goals?
- Deployment: Put the model into production
Originally developed for data mining, it's now widely used across industries for ML projects.
Methodology vs. Notation: Different Levels
| Aspect | Methodology (CRISP-DM) | Notation (GR4ML) |
|---|---|---|
| Purpose | End-to-end process framework | Specific language for requirements |
| Answers | What phases to follow | How to write down requirements |
| Scope | Full ML lifecycle | Business understanding phase |
| Example | "First do EDA, then model" | "Strategic Goal → Decision Goal → Question" |
These are methodologies — end-to-end process frameworks. GR4ML is a modeling notation — a specific language for capturing requirements. They address different levels: CRISP-DM tells you what phases to follow; GR4ML tells you how to write down the requirements within the business understanding phase.
Real-world: IBM, Accenture, and other large organizations have adopted variants of these methodologies, but there is no single universal standard for ML requirements engineering comparable to UML's dominance in traditional SE. Different companies adapt different approaches based on their domain, scale, and ML maturity.
Pitfall: Confusing methodologies with notations. CRISP-DM tells you what phases to follow; GR4ML tells you how to document requirements. They work together, not as alternatives.
Recap: Three major ML methodologies: CRISP-DM (industry standard, end-to-end), SEMMA (SAS, modeling-focused), Big Data Analytics Lifecycle (modern, big data). These are process frameworks, while GR4ML is a requirements notation. They address different levels of the ML development process.
Exam Guidance Summary
Course Progression
The course follows a structured progression through the entire ML system lifecycle:
- Requirements → Architecture → Implementation → Testing → Deployment → Monitoring
- Approximately 10–11 sessions remain to cover the full cycle
Assessment Schedule
| Assessment | Details | Timeline |
|---|---|---|
| Quiz | 5 marks, 15-minute duration, 5-day window | May 22–27 |
| Assignment 1 | 20 marks, one month to complete | Released May 20 |
| Assignment 2 | 20 marks, may include practical lab experiment | Released July 18 |
| Situated Learning | 5 marks, non-traditional format | Post-midterm, TBA |
| Group Formation | Max 4 students per group; individual submission permitted | — |
Webinar Schedule
Four webinars scheduled (7:30–9:00 PM):
- May 19
- June 2
- July 14
- July 28
Topics designed to complement and extend lecture content without overlapping with other courses.
Self-Study Resources
- Apollo autonomous driving case study: Research paper + YouTube video explaining LIDAR, camera, and radar integration
- Microsoft SE-for-ML case study: 9-stage ML workflow and best practices
- GR4ML documentation: Course handouts include links to modeling notation documentation
Upcoming Topics
Scaling and operational monitoring of ML systems (MLOps, LLMOps, AgentOps) are covered in dedicated later sessions (sessions 13–14 include risk awareness and production constraints).
Key Industry Applications
Dominant Industry Patterns
| Pattern | Description | Example |
|---|---|---|
| API-based model integration | ML models deployed as HTTP POST endpoints consumed by larger applications | COVID prediction → hospital management system; credit risk → loan processing |
| Multi-model systems | Real-world ML systems use multiple models working together | 28 models in Apollo autonomous driving |
| Separate SE and ML pipelines | SE (Agile, CI/CD) and ML (exploratory, experiment-driven) run in parallel | Integrate at defined points, not forcing ML into traditional SDLC |
| MLOps as distinct discipline | ML monitoring uses different metrics than traditional app monitoring | Data drift, concept drift, accuracy degradation vs. CPU, memory, uptime |
| Model versioning | Production systems manage multiple model versions simultaneously | ChatGPT progression from 3.5 to 5.5; applications choose version based on capability/cost |
| Embedded ML | Safety-critical systems run models on embedded hardware | NVIDIA Drive AGX, Tesla FSD Computer for real-time, offline-capable inference |
| GR4ML | Goal-oriented requirements modeling for ML applications | Proposed 2021, growing adoption for bridging business goals and ML specs |
| CRISP-DM | Most widely adopted end-to-end ML project methodology | Industry standard across organizations |
SEML Lecture 3 Notes · Requirements Engineering and ML System Architecture
Sections Breakdown
How production systems combine ML models with traditional software infrastructure
Fundamental differences between structured SE and exploratory ML development
The 10 phases of the standard ML pipeline and cyclical lifecycle
Data Scientist, ML Engineer, and Software Engineer responsibilities
Multi-model architecture with 28 ML models and three sensor types
Three triggers: intrinsically hard problems, big data, and time-changing problems
Deterministic vs probabilistic systems and ML-specific requirements
Four goal levels and managing trade-offs between accuracy, latency, and cost
Three-view framework addressing UML's inadequacy for ML requirements
CRISP-DM, SEMMA, and Big Data Analytics Lifecycle
Assessment schedule, webinar dates, and self-study resources
Dominant patterns in production ML systems
Exam Revision Notes
Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.
ML and Non-ML Components
Must-know: ML systems consist of both ML and non-ML components; the model is one piece surrounded by infrastructure (UI, auth, storage, monitoring). ML can be core (speech recognition) or supporting (audit risk prediction).
Top pitfall: Assuming the ML model IS the product. The model is one component surrounded by substantial infrastructure.
Self-check: Name three non-ML components needed for a speech recognition system.
Connects to: SE and ML Process, ML Pipeline
SE vs ML Process
Must-know: SE and ML processes are fundamentally different and run in parallel. SE is structured (Agile, Scrum); ML is exploratory (experiment-driven). Operations differ: SE monitors system metrics (CPU, memory); ML monitors model metrics (data drift, concept drift). Integration is one-time; operation is continuous.
Top pitfall: Assuming deployment is the end. Models degrade over time due to data drift, requiring continuous monitoring and retraining.
Self-check: What is the difference between data drift and concept drift?
Connects to: ML and Non-ML Components, ML Pipeline
ML Pipeline and Lifecycle
Must-know: The 10 phases of the ML pipeline: requirements, data collection, cleansing, labeling, feature engineering, training, evaluation, deployment, monitoring, retraining. The lifecycle is cyclical — retraining loops back. Data and model versioning track lineage.
Top pitfall: Treating the ML pipeline as a one-time process. Production ML requires continuous monitoring and retraining.
Self-check: Name the 10 phases of the standard ML pipeline.
Connects to: SE and ML Process, Roles in ML Systems
Roles in ML-Enabled Systems
Must-know: Three roles: Data Scientist (early phases, model building), ML Engineer (integration, deployment, operations), Software Engineer (non-ML components, APIs, databases). Each has distinct skills and responsibilities.
Top pitfall: Assuming one person can handle all roles. Data science, ML engineering, and software engineering require different skill sets.
Self-check: What are the three key roles in ML-enabled systems?
Connects to: ML Pipeline, Apollo Case Study
Apollo Autonomous Driving
Must-know: Apollo uses 28 ML models with camera, radar, LIDAR sensors. Models run on embedded hardware (NVIDIA Drive AGX). Critical vs non-critical components determine failure handling. ADAS levels 0-6 define autonomy and failure tolerance.
Top pitfall: Thinking of ML as single models. Production systems use multi-model ensembles with complex interactions.
Self-check: Name the three sensor types used in autonomous vehicles.
Connects to: Roles in ML Systems, When to Use ML
When to Use Machine Learning
Must-know: Three triggers for ML: (1) Intrinsically hard problems (language ambiguity, sarcasm, context, cultural nuances; vision), (2) Big data (6.3M Google searches/min, 41.6M WhatsApp messages/min), (3) Time-changing problems (fraud detection). If rules work, don't use ML.
Top pitfall: Using ML when simple rules would suffice. ML adds complexity and cost — only use when necessary.
Self-check: What are the three conditions that strongly indicate ML is appropriate?
Connects to: Requirements Engineering, Goals in ML Systems
Requirements Engineering for ML
Must-know: Deterministic (same input -> same output) vs probabilistic (output depends on data, model version, prompt). ML requirements must address: data quality/availability, model behavior under uncertainty, continuous learning/updates.
Top pitfall: Treating ML requirements like traditional SE requirements. ML adds data, algorithm, and deployment concerns.
Self-check: What are the three concerns for ML requirements engineering?
Connects to: When to Use ML, Goals in ML Systems
Goals in ML-Enabled Systems
Must-know: Four goal levels: organizational (reduce dropout 20%), user (mentor identifies at-risk students), system (track engagement), model (predict dropout probability). Trade-offs: accuracy vs latency, cost vs quality.
Top pitfall: Listing goals without acknowledging conflicts. Must explicitly manage trade-offs between accuracy, latency, cost, and quality.
Self-check: What are the four goal levels in ML-enabled systems?
Connects to: Requirements Engineering, GR4ML
GR4ML Framework
Must-know: GR4ML has three views: Business (what/why), Data Preparation (data), Analytics Design (model). Business View six elements: strategic goal, decision goal, actor, question, indicator, insight. Proposed 2021, addresses UML's inadequacy for ML.
Top pitfall: Using UML for ML requirements. GR4ML captures data, algorithm, and deployment concerns that UML cannot.
Self-check: What are the three views in GR4ML?
Connects to: Goals in ML Systems, Alternative Methodologies
Alternative Methodologies
Must-know: CRISP-DM: business understanding -> data understanding -> data preparation -> modeling -> evaluation -> deployment. SEMMA: Sample, Explore, Modify, Model, Assess. Methodologies vs notations: CRISP-DM tells what phases; GR4ML tells how to document requirements.
Top pitfall: Confusing methodologies with notations. CRISP-DM and GR4ML work together, not as alternatives.
Self-check: What are the six phases of CRISP-DM?
Connects to: GR4ML
Was this lecture useful?
BitsNotes AI Assistant
Subject Notes AssistantConfigure AI Chat
Choose how to access the chatbotSigned in as
Powered by BitsNotes — 20 messages per day. No API key needed. Want unlimited access? Use "Bring Your Own Key" mode.
Sign in to use AI Chat
Get 20 free AI messages per day to ask questions about your lecture notes. Sign in with Google or GitHub — it takes 5 seconds.
Sign In to BitsNotesSwitch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.