Orchestration, Automation, and Version Control for Data Pipelines

# Orchestration, Automation, and Version Control for Data Pipelines

10.1 Recap: Data Transformation, Validation and Testing

10.1.1 The Bronze–Silver–Gold and ELT Layer Patterns

The session opens with a quick recap of last class. Two layer patterns were covered, and both answer the same question from different angles: how do you keep the journey from raw data to trusted data organized?

Pattern 1: bronze, silver, gold (the medallion pattern). Think of it as three shelves in a workshop:

• Bronze — raw ingestion. Data lands exactly as it arrived, nothing more. No cleaning, no renaming, no fixing. The point of keeping a bronze copy is that you always have the original to go back to if a later layer corrupts something.
• Silver — filtered, cleaned, and augmented. This is where the work of making raw data trustworthy happens: duplicates removed, types corrected, missing values handled, extra context joined in.
• Gold — final aggregation. The layer built for analysis and reporting, where numbers are summed, grouped, and shaped for dashboards and models.

Pattern 2: the ELT/BDB pattern (build, and then the ELT three-layer data modeling approach: staging, intermediate, and data mart):

• Staging — data is staged first: copied into the warehouse as-is, mirroring the source.
• Intermediate — data is built into an intermediate model: cleaned, joined, and reshaped into reusable building blocks.
• Data mart — the final assembly: the intermediate models are combined into the specific tables that reports and analyses consume.

The two patterns map onto each other naturally: bronze plays the role of staging, silver plays the role of intermediate, and gold plays the role of the data mart. One pattern names the quality of each layer; the other names the purpose of each layer. Both organize the same underlying idea: raw data in, trusted data out, with every step in between visible and auditable.

10.1.2 The Transformation and Validation Lifecycle

The data transformation, validation, and testing flow runs in steps:

1. Discover the data — find the sources and understand what is actually in them.
2. Map the data — decide how source fields become target fields (the mapping rules).
3. Perform the transformation — design the logic and execute it.
4. Validate and test — prove the transformed data is correct.
5. Document and monitor — write down the rules and watch the pipeline over time.

The toolset is deliberately simple and familiar: SQL is used for validation (counts, sums, joins that expose orphans and duplicates), Python is used for validation (logic that SQL expresses badly), and a version control tool such as Git is used to keep track of everything — code, mappings, and documentation.

10.1.3 The Claims-Processing Exercise

The exercise: a system with many millions of claims, and you must decide what kind of data transformation and testing to perform.

The transformation rules are driven by business logic — for example, payout amounts based on policy type, location, and claim history. The testing side includes:

• verifying every client calculation with a checking application (an independent program that recomputes each payout),
• computing weekly payout totals and monthly payout totals,
• building fraud deduction logic with the hard requirement of no false positives (a legitimate claim must never be flagged as fraud),
• running checks by location and by policy as well as in aggregate,
• and a single target: zero payout error.

Real-world connection. This is the everyday world of insurance fintech: claims engines, payout computation, and fraud scoring run on exactly these patterns, and regulators demand provable correctness. The same structure — business rules, independent verification, totals at multiple granularities, a strict no-false-positive fraud flag, and a zero-error target — appears in payroll systems, healthcare reimbursement, and bank transaction reconciliation, wherever money moves based on computed values.

10.2 Schema and Contract Validation

10.2.1 Schema Validation

Step one of making validation work is verifying the schema — the blueprint of a table or dataset: its columns, their types, and the rules they must obey. Ask the two questions that open the work: Do we have the right schema? And do we have all the data entities and all the attributes? (An entity is a thing we store, like Customer or Claim; an attribute is one of its properties, like CustomerName or ClaimAmount.)

Schema validation covers the whole data structure:

• The naming structure — every object follows the agreed naming standard, so a name tells you what the object is.
• The integrity — relationships between tables hold: no child without a parent, no orphan rows.
• The column names — the attributes exist, are spelled consistently, and mean what they say.
• The data types — an integer column holds integers, a date column holds dates. A float appearing where an integer is expected is a type failure.
• The constraints — the rules attached to the table.

10.2.2 Naming Conventions

Naming conventions come straight from practice: while working in the US at Cisco, every company had its own naming standards and conventions — and the important skill was following one standard consistently once it was chosen. The example given: an index is named with the IDX prefix along with the table name.

Following one consistent standard for tables and indexes makes the schema readable and self-describing: the schema becomes documentation in itself, which is the cheapest documentation there is.

10.2.3 Contract Validation

Step two is validation of the contract — the agreement between the producer and the consumer of a data set. In the example here, the policy (the policy data) is the producer and something consumes the policy; the contract defines what the charges are if the terms are broken and what happens if they are not broken. In other words: the contract pins down the expectations — shape, fields, types, frequency — that both sides agree to, and the penalties (rejected data, reruns, escalations) when those expectations are violated.

Real-world connection. This is the producer-consumer pattern of data engineering: source systems, vendor feeds, and partner integrations all ship schemas that evolve. Financial and healthcare integrations live or die by contract management — a partner feed that silently drops a field can corrupt a month of reporting.

10.2.4 Rule Validation and Data Profiling

Every transformation mapping and every rule needs proper validation — rule validation from source to target. When logic such as GST (goods and services tax), or bad or extra charges, is applied, the rule must be validated as data travels from one place to another. Ask three questions for every row:

1. What is the original source value?
2. What is the target value after the rule?
3. Is anything happening in between that should not be?

There must be no data loss: a customer's name that is Devendra Kumar at the source must still be Devendra Kumar at the target. If a transformation rule accidentally truncates, drops, or reorders values, the source and target no longer tell the same story.

After the transformation comes data profiling as post-confirmation. Profiling was covered earlier, so use it to understand the data — all the attributes, all the columns: distributions, null rates, value ranges, unusual patterns. The profile of the customer or the payer also needs to be perfect, because a profile that looks wrong (a column that went from 2% nulls to 60% nulls, an amount column with a sudden spike) is the earliest sign that a rule broke something.

10.3 Testing the Pipeline: Unit, Integration, Regression, and Performance

10.3.1 Regression Testing vs System Testing

The class was asked an open question: why do you do regression testing in industry, and what is the difference between regression testing and system testing?

The first answer offered was that regression testing simulates the real load the application will face when deployed in production. That was set aside — no. The next answer: "just to ensure we are not breaking anything existing." Perfect, correct.

The rejected answer matters as much as the accepted one. Load simulation (testing how the system behaves under heavy traffic) is a performance concern — it belongs to performance testing, not regression testing. Mixing the two up is exactly the kind of confusion this correction is meant to kill: regression is about change safety, not scale.

10.3.2 Automated Regression in Practice

From the instructor's own consulting work at Cisco: even as a data architect, changes were made to procedures and triggers, tables were archived, partitions were attached and detached as part of the work — and regression testing followed every change. A team in Pune ran the automated regression suite, checking the basic functionality. If the existing functionality broke, there was a problem: you had to go back, revert the change, roll back the change.

Two lessons sit inside this exchange. First, regression testing is automated regression testing — you write your own scripts and tools to run the same checks over and over, because no human will reliably remember to re-check everything by hand after every change. Second, regression applies to data pipelines and data warehousing projects just as much as to applications — a data architect changing partitions, archived tables, procedures, and triggers is changing production behavior, and every such change needs the guard.

10.3.3 Unit, Integration, and Performance Testing

A lot of transformation mappings and rules are created; when you add a new transformation logic, it must not break the existing rules. Performance testing makes sure everything performs better, especially with respect to big data. So the test levels are: unit test, integration test, regression test — and performance testing sits alongside as the scale guard.

The same example, one level up, shows why integration tests exist — the rule is now part of a chain:

10.4 Documentation of Data Models

10.4.1 What Good Documentation Includes

After validation and testing comes documentation. Data models need documentation, and they need version control. Proper documentation includes:

• a lot of diagrams — pictures of the entities, the relationships, and the flow;
• table definitions — every table's columns, types, keys, and constraints;
• usage scenarios — how the model is meant to be used, with realistic examples;
• and database design notes — the reasoning behind the design decisions.

Good documentation makes the system easy to use and easy to modify: it gives clarity (anyone can understand the model) and easy maintenance (anyone can change the model safely). When a new person joins, or a new team takes over a project, they always talk about the application and the database architecture — good documentation serves them at all three levels of understanding:

• Conceptual understanding — what the business objects are and how they relate (the big picture).
• Logical understanding — the entities, attributes, and rules without hardware or storage details (the design).
• Physical understanding — the actual tables, indexes, partitions, and storage choices (the implementation).

10.4.2 The Data Dictionary Habit

The instructor's habit as an architect: while supporting a fintech company and reviewing their migration strategies, the first question asked was always — where is your data model? Do you have any documentation? Where is your data dictionary?

A data dictionary is the single reference for every piece of data: every attribute, every entity, every data frame, every data set, documented with its name, meaning, type, and rules.

Documentation is the key to understanding for future implementation: the fintech migration review starts at the model because every downstream decision — what to migrate, what to transform, what to retire — is only as good as the understanding of what currently exists.

10.4.3 Tools for Documentation and Model Management

Real-world: several tools help here.

• dbt Core — an open source tool that helps create the models and also provides version control: models are written as code, reviewed, and versioned like software.
• DataHub — a tool for building data models and managing metadata, discovering data, and data governance: a central catalog that answers "what data exists, where, and who owns it."
• Amundsen — another tool you can use for data discovery and metadata: finding the right data set, its lineage, and its quality signals.

Real-world connection. Documentation is where data engineering and knowledge management meet: model registries, data catalogs, and dictionaries are standard parts of the modern data stack, and migration projects (like the fintech review) are won or lost on the quality of the model documentation they inherit.

10.5 Version Control for Data, Models, and Pipelines

10.5.1 Why Version Control

Versioning is very, very important, and the class was asked: why do we need version control in the corporate world?

The answer has three parts packed into one idea:

1. State capture — version control records the change in state from one state to another: the version of your code, the version of your PowerPoint, non-code material too.
2. Change tracking — it keeps a track of what is changing, from which state to what state.
3. Attribution — it keeps historical references of who is making those changes, because if you need to track down some kind of conflict, version control helps you trace back.

10.5.2 Debugging, Baselining, and Upstream–Downstream Safety

More reasons were added by other students and the instructor. Version control helps for debugging and maintenance: a client says "the chain you sent is not reflecting in my system", so you check the version installed on that machine versus the master code — baselining is important. A baseline is a recorded, frozen version that both sides can agree on: the client's machine runs baseline X, the master code is baseline Y, and the difference between X and Y is precisely the change history to investigate.

Version control also matters for upstream and downstream applications: if there is a version change in an upstream application, the downstream application must take care of it; if the downstream breaks and is not ready, you can roll back to the previous version. The same safety net that guards a single file guards whole systems: an upstream change that breaks a downstream consumer is undone by reverting to the last known-good version.

Over a career of 30-plus years, everyone uses some version control, one way or another — even a folder of dated files is a crude version control system. The difference between crude and proper version control is whether the history is queryable: can you find who changed what, when, and why?

10.5.3 Versioning ML Models and Data Sets

Most data scientists and data engineers also do version control for machine learning models and deep learning models, and data sets go through changes too — preserving the working version while you are testing has many advantages.

The instructor recalled training banking people in Malaysia about ten years ago, before AI was in use there: customer data, where the data is stored, everything before and after the pipeline — all properly maintained. The lesson predates the AI boom: regulated industries were practicing data versioning long before models needed it, because their compliance obligations demanded the same bookkeeping that ML reproducibility now demands.

10.5.4 Naming Conventions and Version Numbers

Just like table names, file names, and index names, versions need proper naming conventions: which version is the development version, which is the test version, which year does that file name belong to. Software products follow this too — Microsoft drivers show it in their properties: there is a major version and a minor version, a file version and a product version.

The example given is a version number like 3.2.4, which maps each position to a level: three is the product, two is a particular component, and four is the sub-component. Written generally:

\[v = a.b.c\]

where \(v\) is the full version identifier, \(a\) is the major level — the product — \(b\) is the minor level — a particular component — and \(c\) is the patch level, the sub-component.

Data sets can also be versioned by state: incomplete, complete, filtered, unfiltered, cleaned, uncleaned. Version one may be incomplete, version two complete, the next one filtered, then unfiltered — different naming standards and version controls for each. The state labels make the version history self-describing: "data set v3 (filtered)" means something precise, while "data set v3" alone could mean anything.

10.5.5 Versioning Approaches: Duplication, Metadata, Tooling

There are several approaches to versioning data and models.

• File versioning — give each file a proper name carrying the version, the year, and the state: e.g., claims_2024_raw_v1.csv, claims_2024_clean_v2.csv. Simple, human-readable, and works anywhere.
• Tool-based versioning — use version control tools: DVC (Data Version Control), Delta Lake, or even Git for data sets if you have enough storage; it also helps for real-time editing.
• Full duplication — if you want to save the full set as it is, duplicate it: the source version as a full copy in the new location, and then start creating versions on the copy. Expensive in storage, unbeatable in simplicity.
• Metadata-based version control — like the tracking history in an SRS (system software requirement specification) document: record when the document is created; give the particular file name or table name and state that it is valid up to which date, so you control the life cycle of particular data attributes with metadata — when the attribute becomes active, and when it is retired.
• And tooling — use version control tools such as CVS, SVN, Git LFS, or LakeFS for the heavy lifting.

10.5.6 Choosing a Tool: Modality, Diffing, and Comparison

A student raised the distinction between two types of version control: code version control, where ETL code is kept in GitHub or GitLab, and data version control, where it is not source code but a database you can backtrack with time travel.

The scope is deliberately narrow: data, data sets, and ML, touching ML code — not the entire project. Whole-project versioning drags in three larger disciplines — change management (who approves changes), change control (how changes are governed), and configuration management (how the whole environment stays consistent) — which are beyond this lecture's scope.

Another student shared that Delta Lake's time travel feature was implemented in their tables — a real production example of data versioning: the table remembers its past states, so a query can ask "what did this table look like yesterday?" exactly as code version control asks "what did this file look like yesterday?"

10.5.7 The Version Control Landscape

Real-world: Neptune.ai is one such tool for version control and managing metadata. dbt is an open source version control system for machine learning models, data sets, and metrics. Git LFS (Large File Storage) is for large file storage — AWS, Google, and Hadoop use big distributed file systems, and if you already know how to commit, push, and pull with Git, Git LFS is an easy open source option to try. In the old days, SVN and TFS were used; Git is the newer protocol. ClearCase was used around 2010 — one student had implemented UCM (Unified Change Management) with multi-site version control and multi-master replication, with changes moving between locations such as Chelmsford, Salt Lake City, Phoenix, Chennai, Korea, London, and Reading.

Real-world connection. From regulated banking data books to Netflix-style ML pipelines, version control is the shared bookkeeping of machine learning: model registries, data versioning tools, and compliance audits all rely on the same principle — every state is named, every change is attributed, and every version can be revisited.

10.6 The Four Rishis: Knowing When Not to Comment

10.6.1 The Story

A story the instructor used to tell leadership teams, senior data architects, project managers, and developers — about four rishis living in the Himalayas. A rishi is a sage, a seeker of spiritual knowledge. Their goal, like everyone's, was clear: to reach heaven and touch the lotus feet of God, to end the cycle of rebirth. They did tapas (severe spiritual discipline) for many years in the cold.

Then one fine day came the Akashwani — the oracle, the voice from the sky: God had accepted their wish; get ready, tomorrow a devdoot (a messenger) would come. The four prepared themselves. A Pushpak Viman arrived — the flying chariot, like the one that carried Ravana when he kidnapped Sita. The messenger said: congratulations, you have done great work, get ready to fly to heaven — but there is a condition. You may not talk to each other. Each of you sits in a separate direction — east, west, north, south — and there is no guarantee that all of you will go up; it depends on your behavior, how you perform.

The Pushpak Viman rose slowly. It flew over Goa, over beaches, toward Kerala, over forests. It moved slow, and below there was a huge lake. On one side of the lake, a tiger mother had just delivered cubs and was licking water. On the other side, a mother deer had delivered a baby deer and was also thirsty, drinking at the edge. The four watched. In a fraction of a second, the tiger noticed the deer, ran fast, attacked, killed the mother deer, and dragged it to the tiger's side.

The first rishi reacted immediately: what the hell did this tiger do? It just delivered babies; the deer is also a mother. How can this happen? This is Pabam, Anyayam — sin, injustice. He was dropped from the Pushpak Viman.

The second rishi: what the tiger did is right. A tiger doesn't eat grass. Deer are meant to be killed. No problem — what is the big deal? He too was dropped.

The third rishi was gossiping — commenting about what the first and second were saying: the first lost his mind, the second lost his mind, everything is known to God, what are they talking about? The third was also dropped.

The one person who did not say a word stayed quiet — and only he went up.

10.6.2 The Twist: The Messenger's Questions

The story has a twist. The messenger asked the person on the east side: what do you think? He said the tiger did wrong — dropped. The person on the west side, thinking the first one was wrong, said: the tiger did right — dropped. The third said: it could be 50% right, 50% wrong — maybe true, maybe false — dropped.

10.6.3 The Lesson for Work and Life

Sometimes it is better to stay on that boat. In the industry, things happen that we do not know why — one good person may be working very hard and still be laid off. The instructor, who has done restructuring and laid off many people, said: it is okay to feel pity for them; it is okay not to feel pity. But talking in the middle — that is gossip. When we don't know, we avoid gossip. If such a conversation happens — "I don't know why your organization is doing this, why my manager is doing this" — either you don't know, and you leave it to the one above (Uparwala).

10.7 Orchestration and Automation of Data Pipelines

10.7.1 From Workflow to Orchestration

We are building machine learning process pipelines and data pipelines, and we define a workflow. First question: what is the difference between a workflow and a design? Workflow is used in many ways — in an automation industry (the instructor worked at Applied Materials), a workflow is like a recipe: follow the steps, and finally you get an outcome; any specific order follows a workflow. In a machine learning pipeline, data goes through some process and finally we get the final data. Asked whether it is good to automate the workflow or orchestrate it — certainly, yes. One student's phrasing captured it: workflow is how it needs to happen.

10.7.2 Why Manual Pipelines Fail

The scenario: you are a data engineer with lots and lots of reports. Extract from multiple sources, transform, load into a warehouse, generate reports with a reporting tool — Power BI, Tableau, or Python. There are new channels, mobile applications, the third wave of BI — new types of experience, more customers, new insights at new frequencies. Not like earlier: daily or weekly replication. Now the frequency is hourly, every two hours. All of this pushes us toward automation and orchestration: many data sources, many transformations, lots of ETL, and finally reporting and visualization — while accommodating new upstream channels, more customers, more report types.

On that last point: the instructor recalled 24x7 production support in the US, building data marts and data models and reports for mutual funds. When an error came, fix it, make sure the reference data was loaded, then wait a couple of hours for the next flow — set an alarm, wake up in two hours, wait for other systems to push data or complete. Lots of waiting, and humans get tired. The solution is data pipeline orchestration.

10.7.3 Data Pipeline Orchestration Defined

10.7.4 Data Orchestration vs Data Pipeline Orchestration

10.7.5 What Orchestration Buys You

We need data orchestration for efficiency, reliability, scalability, and flexibility — and we should be able to monitor and visualize.

Real-world: an e-commerce company or a telecommunication company gets real-time analytics from pipeline orchestration: continuously ingest data, move data, process data, and get insights — more importantly, detect anomalies. It enables data integration across applications — multiple applications, multiple systems. Machine learning and deep learning models can automate pipelines: orchestration enables the integration of ML and AI pipelines — data pre-processing, feature engineering, model training, and model serving — automating the end-to-end AI workflow. It also supports data governance and compliance — HIPAA for healthcare, GDPR for payments — many regulations can be met by using proper orchestration.

10.8 Orchestration vs Choreography

10.8.1 The Two Concepts

The student's first thought (music vs dance) was the trigger, not the answer: it is a useful distinction of vocabulary, but the real difference runs deeper. Both describe a sequence of steps — but the two differ on who is in charge:

10.8.2 Loose Coupling and Testability

A student asked whether the broker architecture is similar to master-slave, where a master is responsible for all the interactions between the slaves. Not exactly, but close: there may be multiple slaves and multiple brokers — but an orchestrator is just one. If there are multiple orchestrators, then yes, it is like multiple coordinator agents; with one master and multiple slaves, the master can be the coordinator agent, the orchestrator. For a very complex pipeline there may be two orchestrators: one main brain — like A.R. Rahman the main conductor — with one orchestrator taking care of the lights and control, another taking care of the instruments. Generally speaking, the broker architecture pattern is exactly what we are talking about: everything is streamlined through the orchestrator.

Visual intuition. Picture two pictures side by side. Left: point-to-point — five boxes (A, B, C, D, E) with arrows between nearly every pair; the drawing is a tangle, and a change to any box ripples through every arrow. Right: orchestrated — the same five boxes on the edges, one broker box in the center, and each service has exactly one arrow: to the broker. The center is the only busy node; the edges stay clean. One-sentence takeaway: orchestration trades a tangle of direct links for one central coordinator — which is why couplings stay loose and tests stay local.

10.8.3 The Scope of Orchestration

10.8.4 Exam Expectations

With the exam target in view, the vocabulary itself needs one more contrast — the difference between a pipeline and a workflow — which is where the session goes next.

10.9 Data Pipeline vs Data Workflow

10.9.1 The Comparison

The professor's water analogy, extended. One pipe carries water from a tank to a tap: that is the pipeline — a single, simple, mechanical flow. But the city's water supply involves the tank, the corporation, the bore well, the motor, pressure valves, and the decisions about who gets water when: that is the workflow — the whole process of dealing with the data, including the parts that are not pipes at all.

10.9.2 The Pipe and Filter Connection

You can relate the workflow to the pipe and filter pattern — the instructor shared links to study both the broker architecture pattern (10.8) and the pipe and filter pattern, calling them very important and very helpful for building data workflows.

10.10 Workflow Orchestration and Time Dependencies

10.10.1 Coordination, Compliance, and Governance

Workflow orchestration means coordinating tasks across different teams or different systems; compliance and governance are major concerns. Orchestration coordinates and manages the execution of tasks within the pipeline — not just their scheduling, but their ordering, their handoffs, and the records that prove each step ran when it should. That record-keeping is what compliance demands: an auditor asks "did the check run before the data reached the report?" and the orchestration history answers it.

10.10.2 Time Series Dependencies

This is where time series forecasting comes into the picture — very important when you are talking about pipeline workflow. Models like ARIMA, SARMA, and LSTM (long short-term memory) are relevant.

What the time-series models are for. When a slice is missing, forecasting fills the picture: ARIMA (Autoregressive Integrated Moving Average) models a value as a weighted function of its own past values and past errors; SARMA (Seasonal ARMA) adds the repeating seasonal pattern — traffic at 8 a.m. looks like yesterday's 8 a.m., not like 3 a.m.; LSTM (long short-term memory) is a neural network variant that learns to remember and forget patterns over long sequences. All three answer the same operational question: given the slices we have, what should the missing slice plausibly look like? They are the mathematical bridge across the gap — never a substitute for the alert that a sensor is down.

Real-world connection. Traffic control centers, IoT platforms, and stock-market data feeds all run this exact pattern: 5-minute (or faster) data buckets, missing-slice detection, and forecast-based gap handling. Telemetry pipelines in manufacturing and energy monitoring are built the same way — which is why the professor ties pipeline orchestration to time series thinking so firmly.

10.11 Directed Acyclic Graphs and Workflow Core Concepts

10.11.1 What is a DAG

10.11.2 Core Workflow Concepts

10.11.3 Acyclic vs Cyclic

Compare two graphs: the second one has a retry; the second is cyclic, the first is not. Acyclic — there should be no circular dependency: a task cannot depend on itself, nor can it depend on a task that ultimately depends on it. Pipeline graphs must also be acyclic — the graph must not link to a previously completed task, because that would mean the pipeline could run endlessly and never finish the workflow.

Visual intuition. Picture two diagrams. Left: acyclic — four boxes: Task A on top, Task B and Task C below it, Task D at the bottom; arrows run A → B, A → C, B → D, C → D. After A completes, B and C run; when both complete, D runs. Every arrow points downward; the flow always progresses. Right: cyclic — the same boxes but with an extra arrow looping from D back up to A, so a "run" of the pipeline never reaches a finish line. Both are drawn with the same boxes and arrows; the single upward loop is what separates a pipeline from a treadmill. One-sentence takeaway: in a pipeline graph, arrows flow one way — toward completion, never in a circle.

10.11.4 PERT, CPM, and State Transitions

If you studied PERT and CPM models (mechanical engineering, project management), revise them for understanding: earliest start, latest start, earliest finish, latest finish, float, slack — these are graph concepts applied to scheduling.

Scheduling numbers only make sense when the state of each job is known — which brings in the second graph concept the lecture paired with PERT/CPM.

10.11.5 Graphs, Nodes, and Edges

A graph is a structure consisting of nodes and edges — nodes are the things (tasks), edges are the connections between them (dependencies). Real-world: nowadays people use Neo4j on the backend — the instructor taught Neo4j and trained students on it; it is a graph database, very beautiful. Many graph databases exist — people are moving toward MongoDB on one side, and they also use graph databases to store different relationships.

From the shape of the graph, the session now moves to what lives inside it.

10.12 Tasks, Operators, Sensors, and Dependencies

10.12.1 The Sales Data Workflow

A simple example: a start operator, sales data, an operator — then determine the load type. Once the load type is determined, you can define, skip, or continue the sales data load. There is a program for the sales data load and a sales data transformation: we start, we extract the sales data, we transform the sales data, we load the sales data; once the load is done, we go into sales data reporting — prepare the report, publish the report, bingo, done. One simple workflow task.

Start → ExtractSales → DetermineLoadType → LoadSales → TransformSales → PrepareReport → PublishReport

10.12.2 Tasks and Operators

10.12.3 Sensors

Sensors are a special subclass of operators which are all waiting for something to happen — not the IoT sensor, but a trigger, like an Oracle trigger: waiting for some event. It can be time based, or waiting for a file — a kind of dependency: file success.

10.12.4 Dependencies: Upstream and Downstream

Dependencies build the relationship between one job and another job. For example, extra data is the dependency for the internal API load incremental — those are the dependencies; it is like an edge to a node. The key part is how tasks relate to each other — upstream and downstream tasks — that is what dependencies are.

10.12.5 Declare Tasks First, Dependencies Second

10.13 Executors, Schedulers, Logging, and Retries

10.13.1 Executors and Schedulers

When a task is created, the executor is the mechanism by which task instances run — think of objects as instantiated classes. The DAG definition is the class: a blueprint. The task instance is the object: a concrete execution of that blueprint on a particular day, with particular data. At runtime there is an executor and a scheduler.

10.13.2 Logging and SLA

Logging: we have to log every task. Look for the SLA — the service level agreement: at what time it has to complete — and retry: we want to have a retry if some job does not run.

10.13.3 Retries and Loaders

This connects to database loaders. In Oracle, Informix, and many other databases there is the SQL loader, a high performance loader. Loaders report errors, records inserted, records deleted, records skipped. Sometimes you do not have a good data set; you load and duplicates get eliminated, so you rerun the load. One student was rerunning millions of records that very evening. Filters produce accepted records and rejected records; you may send the rejected records to somebody to look at why — it may be an anomaly, it may be an outlier — then you fix and rerun, or talk to the upstream people to fix it, and rerun.

10.13.4 The 32 KB Story

A concrete example from the same evening: a job was created to ingest data; anything that goes beyond 32 KB was treated as some kind of heavy data and was not ingested.

Real-world connection. Every production data platform runs this exact loop: scheduled loads, SLA alarms, loader reports, rejected-record queues, and automated retries. The 32 KB story is a small, perfect specimen of why orchestration exists — a 0.0011% edge case that, without automation, means a human re-running millions of records by hand.

10.14 Orchestration Tools

10.14.1 What an Orchestration Tool Does

What is the purpose of an orchestration tool? It helps you transform data from one to another and achieves the outcome by stringing together the network of tasks, so you do not need to worry about it. Like a website building tool — Figma came up as an analogy — a graphical tool that gives you the luxury to create and load. A student read out the five bullet points: the purpose of a data orchestration tool is scheduling the task, starting the task, saving the states, monitoring the workflows, and generating alerts.

10.14.2 Operator-Native and Container-Native Tools

The data orchestration tool should be operator native: it should have predefined templates — a filtering template, a sourcing template. Power BI is a great example of connectors: data can come from AWS, from a URL source, from JDBC, from ODBC, from CSV, from sensors and real-time streams — thousands of connectors, even MongoDB — JSON files, Excel, XML, APIs. Whatever tool you choose, it needs proper templates to connect to different sources. Tools should also be container native: depending on what you carry — a web service or a JSON file — it can be orchestrated through Kubernetes or any other tool. Microsoft people use AKS (Azure Kubernetes Service) to manage Kubernetes: the infrastructure is managed by Microsoft, and you build whatever image you want inside the package and provide the main image.

10.14.3 The Tool Landscape

Real-world: there are data specific workflow tools and general purpose workflow tools: Kubeflow, Argo, Tekton, AWS Step Functions. Apache Airflow supports dynamic pipeline generation, builds scalable architecture using DAGs, schedules tasks, and automates the ETL process — Netflix is using Airflow. There is Prefect, a Python-based workflow tool; Dagster; Argo — another tool; Kubeflow, where you can build ML workflows. On AWS, the managed service for Airflow is called MWAA (Managed Workflows for Apache Airflow), available through the AWS Academy Data Engineering account.

10.14.4 Evaluate, Don't Idolize

RapidMiner is another good one — the paid tool can be difficult to buy, but a downloadable version is available; it was bought by Siemens. RapidMiner has operators for everything: data import, data access, cleansing, modeling, scoring, validation — validation operators, utility operators, modeling operators — you can add your own operator, apply a model, filter. For people who are not yet working on machine learning projects (some are on Microsoft, some AWS, some Databricks), RapidMiner and Orange are recommended as easy-to-use tools to build rules, transformations, ETL, and pipelines from start to end.

10.15 Designing an End-to-End Pipeline Workflow

10.15.1 The Design Steps

When you want to design a workflow: decide the data source — an IoT device, for example; how you are going to collect and save it — data ingestion; and what kind of validation to build — data type correct, duplicates, are all the columns present. Then you build the transformation of data, model training, and model deployment.

10.15.2 Triggers, Alerts, and the Human in the Loop

You can build a lot of triggers: as soon as data is collected, you start — the trigger event is "a new data file is detected". If validation fails, you create an alert; the alert is sent to the person who is building it; they validate the data; and execution stops — human in the loop.

10.15.3 Logging and Continuous Monitoring

Ingestion runs first, then validation, transformation, model training, and deployment. Along the way you find out what components are missing, what data attributes are missing, what the dependencies are. Log it: the logging file enables continuous monitoring.

Real-world connection. IoT telemetry pipelines (the lecture's own example), clickstream ingestion, and ML feature pipelines all follow this exact skeleton: event trigger on arrival, validation at the door, human alert on failure, and full logging for monitoring — the same structure the orchestration tools of 10.14 are built to express.

10.16 Orange Miner: Building a Pipeline Visually

10.16.1 The Cutlets Dataset

A sample data file was loaded — cutlets, a time series data set with 35 instances, 2 features, no missing values, and no target variable. From the file you can build a data table.

10.16.2 Validation and Transformation

For validation, you can click column statistics and see the statistics of every column. For transformation, you can select only one column, or select a particular row — you decide what column you want, choose the good data, then merge data from another file. From the merged data, perform operations: select unique values only — no duplication — and from the unique values build a graph; visualize the distribution. Everything is covered, from ingestion to visualization.

10.16.3 Pre-processing and Python

Pre-processing is available too: data imputation — find the missing values and apply imputation rules. There is also a Python script widget: the Python script can perform an operation and give an output, and that output can be a data table.

10.16.4 Models and Versioning

You can build models: once the data table is created, apply a logistic regression or an AdaBoost — the data table goes into AdaBoost, and AdaBoost gives a result. Save the model — like the pickle file in Python: save model, load model. That saved model is a version: the entire thing is saved, the version is there, and you can use it later for loading that model.

Real-world connection. No-code/low-code tools like Orange and RapidMiner are the on-ramp to data engineering and ML: analysts prototype pipelines visually, business teams explore data without code, and the same pipeline shapes carry over to production tools like Airflow and Kubeflow when the scale demands it.

Exam Guidance Summary

Everything below appeared in the session as explicit exam intel. The full explanations live in the sections referenced.

• Orchestration (10.8): the final comprehensive examination may ask about orchestration — what types of orchestrations exist, why you orchestrate, and your best practices in implementing orchestration. Be ready to explain the why and the best practices, not just the definition.
• Build-your-own workflow (10.12): expect a question where you create your own workflow — a pipeline workflow given some example.
• DAG build order (10.12): when building a DAG, remember the procedure: declare the task first and then declare the dependencies second; first declare all the tasks — get that order right and the DAG builds itself.
• PERT and CPM (10.11): revise these concepts: earliest start, latest start, earliest finish, latest finish, float, and slack — these are graph scheduling concepts that help you understand workflows.
• Pipeline exercise (10.11): a pipeline-building exercise is coming: you will get one pipeline exercise and you will build one pipeline.
• Comparisons to be ready for (10.3, 10.7, 10.9): be ready to contrast regression testing with system testing; unit vs integration vs regression vs performance testing; data pipeline vs data workflow; and data orchestration vs data pipeline orchestration — all of these were discussed in depth in this session.

Key Industry Applications

The session grounded every concept in a named real-world setting. The full stories live in the sections referenced.

• Claims processing (10.1): payout amounts by policy type, location, and claim history; weekly and monthly payout totals; fraud deduction logic with a no-false-positive requirement and a zero payout error target.
• Cisco (10.3): automated regression suites run after every change to procedures, triggers, partitions, and archived tables — roll back when basic functionality breaks.
• Fintech (10.4): migration reviews start with the data model, documentation, and data dictionary.
• Model documentation tools (10.4): dbt Core, DataHub, and Amundsen for data model documentation, metadata management, and data governance.
• Data and model versioning (10.5): Delta Lake time travel in production; Git LFS for large files; Neptune.ai for ML metadata; DVC for data set and model versioning.
• Multi-site version control (10.5): ClearCase UCM with multi-master replication; FAA compliance reporting between versioned reports.
• Real-time analytics (10.7): e-commerce and telecom: real-time analytics, continuous ingestion, and anomaly detection through pipeline orchestration.
• Compliance via orchestration (10.7): healthcare (HIPAA) and payments (GDPR) compliance supported by proper orchestration.
• Traffic IoT (10.10): signal data pushed every five minutes to a central monitor; time series models (ARIMA, SARMA, LSTM) for handling missing time slices.
• Graph databases (10.11): graph databases such as Neo4j store workflow relationships.
• Loaders and reruns (10.13): Oracle SQL loader and high performance loaders for rerunning failed loads — 33 records of 3 million skipped for exceeding 32 KB were re-ingested automatically.
• Orchestration tools (10.14): Apache Airflow (used by Netflix), Prefect, Dagster, Argo, Kubeflow, Tekton, AWS Step Functions, and AWS MWAA as orchestration tools.
• No-code pipelines (10.16): Orange and RapidMiner as no-code/low-code tools for building ML pipelines visually.