Imitation Learning

17.1 What Imitation Learning Is and Why It Matters

17.1.1 Definition in Plain Words

Imitation learning — learning by copying an expert — is a way to teach an agent what to do by showing it what a good performer does, rather than by writing a reward by hand and letting the agent discover a policy through trial and error alone. The word imitation is common outside machine learning. We learn many skills by watching others, sometimes even picking up habits like how a manager handles a situation, and later we become our own best version — the lecture used the phrase "behave like your boss" to make this concrete: you observe a skilled manager, you copy responses to clients, to deadlines, to conflicts, and over time you distill your own style. The same idea applies here. The learner, the agent that needs to act, gets access to a demonstrator or to demonstrations, which are records of how that demonstrator behaved.

An expert is someone or something that is good at the task, and expert demonstrations are the data that expert leaves behind. A demonstration is not a single label — it is a set of state-action pairs logged while the expert acts. Access to the demonstrator can mean full trajectories, or just the ability to ask the expert what it would do in a given situation. When we have many such records, we can treat imitation as supervised learning on those records, at least as a first step.

17.1.2 Two Roles: Learner and Expert

There are always two sides. One side is the learner that is trying to acquire a skill. The other side is the expert whose behavior is worth copying. We assume the expert is skilled, and we want the learner to become as close as possible to that skill level. This framing already hints why imitation is useful: when it is hard to write down the right reward, but it is easy to show what good behavior looks like, imitation gives a direct path.

Think of it like learning a sport or a workplace routine. At first you watch and copy, and that helps you get started. Over time you may need more than copying, but as a starting strategy copying is powerful. That is the core motivation that runs through the whole session.

To make the roles precise:

• Learner: the agent under training, with parameters $\theta$ that define ${\hat{\pi }}_{\theta }$. It starts weak and improves by minimizing a gap to the expert.
• Expert: the reference performer, denoted ${\pi }^{\ast }$. The expert may be a human, an existing controller, or even a stronger RL agent. We do not need the expert's internal reward — only its surface decisions.

The learner's goal is not to memorize exact trajectories but to generalize the mapping so that for a new state it still picks an expert-like action. Imitation learning is therefore most valuable when expert data is cheaper than reward engineering — for example, asking a pilot to fly a demo is often cheaper than encoding "good flying" as numbers.

Visual intuition: picture two columns. Left column lists expert states — a camera frame of a straight road, a frame of a left bend, a frame of a crowded crossing. Right column lists the expert's hands on the wheel for each frame. A line connects each frame to its action. The learner's job is to draw the same lines for new, unseen frames.

17.1.3 How This Topic Sits in the Course

Exam note: Earlier material on TPO loss functions and their numerical details was set aside as not in scope for this course and not needed for the exam. The focus instead is on topics that are part of the syllabus and will appear in the assessment, and imitation learning is one of those topics. The treatment here stays at the level needed for understanding the idea, its strengths, its failure modes, and the key algorithms, without going into low-level loss-function numericals that were covered only for completeness.

Real-world: The same shift happens in practice. Teams often start with a hand-tuned reward, then realize that showing examples from a human driver, a game player, or a pilot is far easier than writing the perfect reward. In warehouses, a hand-coded reward for "pick neatly" is brittle, but a few hours of human picking demos give a solid starting policy.

17.2 From States and Actions to Supervised Examples — The Track Intuition

17.2.1 State-Action Pairs as the Basic Unit

A demonstration means watching someone act in a real environment and recording what happened. At each moment we note the state, written $s$, which is the situation the agent faces, and the action, written $a$, which is what the expert did in that state. A pair $(s,a)$ is a state-action pair. A collection of many such pairs is the training material.

The verbal idea in the lecture was "if someone is doing something right, you talk about state-action pairs — what does that expert do in each state." The reconstructed idea is that a trajectory is a sequence of states together with the expert's choices, and we treat each step as an example for supervised learning.

A single demonstration already contains many pairs because the expert visits many consecutive states. The same state variable $s$ may represent a raw image, a sensor vector, or a board position, depending on the domain. The action $a$ may be a steering angle, a move, or a control command. What matters is the pairing: "in this situation, do this."

17.2.2 The Racing Track Thought Experiment

Picture a car on a small racing track. The task is to steer — go straight, turn slightly left, or turn slightly right — depending on where the car is on the road. If the whole expert drive were drawn on the track, there would be a line that shows where a skilled driver would place the car at each point. Each short segment of that line can be thought of as a state: the position and heading of the car at that moment. For each such state there is a correct steering action, drawn in the lecture as a mark in a different color — red for the state location, green for the action.

Even a small track generates many states. Each lane position, each slight shift in angle, is a different input. If we record enough of those states together with the expert's steering choice, we obtain millions of examples for a small track. We can then feed those examples to a deep neural network and ask it to learn the mapping: "given this track view, what should I do?"

Formally, a policy is a mapping from states to actions, written

$$\pi :\mathcal{S}\to \mathcal{A}$$

where $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, and $\pi (s)\in \mathcal{A}$ is the action the policy suggests when it sees state $s$. The state variable $s$ lives in $\mathcal{S}$, the action variable $a$ lives in $\mathcal{A}$, $\pi$ is the function we want to learn, and $\pi (s)$ is its output at $s$.

The hope is simple: take state-action pairs from the expert, treat them as supervised labels, and train $\pi$ to reproduce them.

Visual description: draw a top-down oval track. Mark the expert path as a continuous blue line along the center. At evenly spaced points along it, place a small red dot (the state) and a short green arrow (the action — straight arrow on straights, left-curved arrow before a left bend). The lecture used red for location and green for steering to separate "where you are" from "what to do." Zooming in, even a 10-meter straight yields dozens of slightly different camera views — each is a distinct $s$, each needs its own green arrow.

17.2.3 Why the Supervised Reduction Is Tempting and Incomplete

On the surface this reduces reinforcement learning to supervised learning: states become inputs, expert actions become targets, and a network learns to predict the target. That is attractive because supervised learning is well understood.

The limitation appears right after that. No matter how many pairs we collect, we cannot cover every possible state and every possible action. More importantly, supervised learning treats each state independently, as if the choice now has no effect on what state comes next. In a real task the choices form a sequence that achieves a goal. One steering decision changes the next camera view, which changes the next decision. When each prediction is made in isolation, the learner misses that sequential dependence.

This mismatch between "treat each example alone" and "actions form a sequence that creates the next state" is the seed of the failure that appears later with cloning.

17.3 What You Need to Run Imitation Learning

17.3.1 The Ingredients

If you want to learn imitation with a supervised network, you need several pieces working together:

• a policy class, which is the family of functions you allow — typically a deep neural network — written $\Pi$ for the set of all permissible policies, and a specific learned policy ${\hat{\pi }}_{\theta }$ inside that set with parameters $\theta$;
• a loss function that measures how far the learner is from the expert;
• a learning algorithm that updates $\theta$ to reduce that loss;
• a demonstrator or a set of demonstrations that supply state-action pairs;
• and a practice environment, either a simulator or the real world, where the learner can try its policy and gather new states.

Here $\theta$ denotes the network weights — the numbers the network learns — and ${\hat{\pi }}_{\theta }(s)$ is the action the network predicts at state $s$ when it uses those weights.

Other pieces in one line each:

• Loss $\mathcal{L}$: a number that is small when ${\hat{\pi }}_{\theta }(s)$ matches ${\pi }^{\ast }(s)$ and large when they differ (squared error is the running example).
• Learning algorithm: usually gradient-based optimization, repeatedly stepping $\theta \leftarrow \theta -\eta {\nabla }_{\theta }\mathcal{L}$.
• Demonstrator / expert ${\pi }^{\ast }$: the source of labels. Access may be offline (a fixed log) or interactive (you can query ${\pi }^{\ast }(s)$ for a new $s$).
• Practice environment: where rollouts happen. Without it you cannot see what states your current ${\hat{\pi }}_{\theta }$ would actually visit.

17.3.2 Where These Pieces Meet

The network takes a scenario $s$ as input and produces an outcome $\hat{a}={\hat{\pi }}_{\theta }(s)$. The expert recommendation for that same scenario is $a^{\ast }={\pi }^{\ast }(s)$, where ${\pi }^{\ast }$ denotes the expert policy. The loss measures the gap between $\hat{a}$ and $a^{\ast }$. Learning means choosing $\theta$ so that gap is small on average over the states the expert visits.

In symbols, for a single state the gap is $\mathcal{L}({\hat{\pi }}_{\theta }(s),{\pi }^{\ast }(s))$, and training aims for

$${\mathbb{E}}_{s\sim {\rho }_{{\pi }^{\ast }}}\left[\mathcal{L}({\hat{\pi }}_{\theta }(s),{\pi }^{\ast }(s))\right]\text{ small}$$

where ${\rho }_{{\pi }^{\ast }}$ is the distribution of states encountered when the expert acts (defined formally in 17.5). The expectation says "average over the kinds of states the expert tends to visit."

Real-world: This is the same stack used when training a lane-keeping network from human drives. The network is a convolutional net, the loss is mean squared steering error, the optimizer is gradient descent, the demonstrator is a human driver, and the practice ground is either logged video or a driving simulator.

Visual: imagine a block diagram with five boxes — Policy Class, Loss, Optimizer, Demonstrations, Environment — arrows show demonstrations feeding $(s,a^{\ast })$ into loss, environment feeding new $s$ to the policy, and optimizer feeding updated $\theta$ back to the policy.

17.3.3 Scope for Assessment

Exam note: The focus for the exam is on understanding these blocks conceptually and on being able to walk through the algorithms that use them, not on reproducing low-level loss numericals. Slides that show detailed numbers exist for reference, but the assessment emphasis is on the ideas and their trade-offs.

17.4 Early Success Stories — Autonomous Driving, Ghosting in Games, and Helicopter Acrobatics

17.4.1 NAVLAB and the Earliest Autonomous Driving Work

One of the oldest examples in the field is the NAVLAB effort at Carnegie Mellon, where a simple neural network learned to drive by using an expert nearby. A truck or car learned to steer on a track from human steering examples, following the exact pattern described earlier: many state-action pairs, a network that maps track view to steering, and an expert who shows what to do. This was presented as a two-minute clip to convey how a very simple network, with an expert around to demonstrate, could already learn autonomous driving on a confined road. It is noted as one of the first works that can be called imitation learning in the reinforcement learning community.

Details that make the pattern concrete: the input $s$ was a downsampled road image, the target $a$ was a steering angle, the training set was a few minutes of human driving recorded as thousands of pairs. At test time the network ran live on the same road and produced steering in real time. Performance was limited to the road type seen, but on that road it stayed centered — an early proof that imitation can replace manual control rules.

Real-world: The same data pattern — camera view paired with human wheel angle — still powers modern lane-assist datasets, only with larger networks and much more data. Modern extensions add multiple cameras, lidar, and DAgger-style corrections, but the core $(s,a)$ logging remains.

17.4.2 Ghosting in Team Games

A second illustration comes from team sports such as soccer and basketball. Every past game is a trajectory: players move, the ball moves, and the sequence shows different ways to approach the same situation. If you record many games, you can study how an average player or a strong player moves when the ball is in a given spot.

Ghosting is a technique that uses that history. When a live game is underway, each dot or number on the display can be a ghost — a replay of how a typical or expert player moved from that same game situation. That ghost is superimposed on the live players. If the ghost behaves differently from what is actually happening, it signals that the current play is deviating from the historical norm. That deviation itself can be studied, and a policy can be learned from the actual moves that players took.

The lecture linked a short clip to make this concrete, noting that audio is not essential — the visual of ghost trajectories laid over live play already shows the idea: multiple trajectories from different players, many games, and a learned sense of what usually happens next.

Real-world: Broadcast tools now overlay expected-run paths in soccer and expected movement in basketball for strategy review. The same learned trajectory models support opponent modeling and training planners that ask "what would a good player do here?" Clubs use these overlays to design pressing triggers and to spot players who consistently beat the ghost.

17.4.3 Helicopter Acrobatics from Demonstrations

A third example is learning difficult helicopter maneuvers. Research from Stanford, including work co-authored by Andrew Ng, showed a helicopter learning acrobatic moves from expert demonstrations. The demonstrations contain the hard-to-engineer details of how to sequence controls for flips and sharp turns. Instead of writing the perfect reward for acrobatics, the system learns from traces of how a skilled pilot or an existing controller flew.

Real-world: This pattern appears wherever perfect reward design is hard — surgical robots that copy a surgeon's path, or robotic arms that learn assembly from human demonstrations. In each case the expert trace encodes timing and compliance that a hand-written reward would miss, and imitation bootstraps a viable policy before any RL fine-tuning.

Visual across all three: picture three panels side by side — left panel: road image with steering arrow (NAVLAB); middle panel: soccer field with translucent ghost dots over live players (ghosting); right panel: helicopter silhouette mid-flip with control traces below (acrobatics). The common arrow is "expert trace → supervised pairs → learned policy," with complexity rising from left to right.

17.5 Behavior Cloning — Cloning an Expert with Supervised Learning

17.5.1 The Simplest Form of Imitation

The simplest form of imitation learning is behavior cloning. The name says it: clone someone's behavior by supervised learning. You collect state-action pairs from the expert and train a network to predict the expert's action whenever it sees the same state.

Formally, suppose you are given a distribution of states visited by the expert, written ${\rho }_{{\pi }^{\ast }}$ or $s\sim {\rho }_{{\pi }^{\ast }}$, and for each such state the expert's action $a={\pi }^{\ast }(s)$. You want to learn a policy ${\hat{\pi }}_{\theta }$ that is close to ${\pi }^{\ast }$ on those states and that also generalizes to states you have not seen.

17.5.2 Mathematical Formulation

The lecture stated the goal as "learn the network parameters $\theta$ such that the expected loss is minimal, given state-action pairs as taken by the expert on those states the expert demonstrated, so the agent performs as good or as close to the expert and generalizes beyond seen states." The reconstructed math preserves that verbal description alongside the expression.

Let $s\in \mathcal{S}$ be a state, $a^{\ast }={\pi }^{\ast }(s)\in \mathcal{A}$ be the expert's action at that state, ${\hat{\pi }}_{\theta }(s)$ be the learner's prediction with parameters $\theta \in {\mathbb{R}}^d$, and $\mathcal{L}(\cdot ,\cdot )$ be a loss that compares two actions. Then the behavior cloning objective is

$$\hat{\theta }=\arg \underset{\theta }{\min }{\mathbb{E}}_{s\sim {\rho }_{{\pi }^{\ast }}}\left[\mathcal{L}\left({\hat{\pi }}_{\theta }(s),{\pi }^{\ast }(s)\right)\right]$$

where ${\mathbb{E}}_{s\sim {\rho }_{{\pi }^{\ast }}}$ is the expectation — the average — over states the expert visits, ${\rho }_{{\pi }^{\ast }}$ is that visit distribution, and $\mathcal{L}$ is small when the two actions match. In words: pick weights $\theta$ that make the average gap between learner and expert, measured on expert states, as small as possible.

A common choice for $\mathcal{L}$ mentioned as "pretty straightforward" and not to worry about, is squared error. The verbal description was "take the difference and square it; you might want to measure the squared distance or mean squared error between your action and the expert's action." That reconstructs to

$$\mathcal{L}(s)={\Vert {\hat{\pi }}_{\theta }(s)-{\pi }^{\ast }(s)\Vert }^2$$

or in the scalar steering case

$$\mathcal{L}(s)={\left(\hat{a}(s)-a^{\ast }(s)\right)}^2$$

where $\Vert \cdot \Vert$ is the Euclidean norm — the straight-line length of a vector — and the square makes large errors count more than small ones. For discrete actions the same idea uses cross-entropy, but the squared-error form above is the one emphasized in the lecture.

At training time the network receives a scenario $s$, produces $\hat{a}$, compares it to the expert label $a^{\ast }$, computes the loss above, and updates $\theta$ to reduce it.

Derivation of the empirical loss: the expectation is not computed analytically; it is approximated by the sample average over the demo set

$$\hat{\theta }\approx \arg \underset{\theta }{\min }\frac{1}{N}\sum _{i=1}^N\mathcal{L}({\hat{\pi }}_{\theta }(s_i),a_i^{\ast })$$

so minimizing the familiar mean squared error on the demo set approximates minimizing the population objective over ${\rho }_{{\pi }^{\ast }}$. As $N$ grows, the sample average converges to the expectation under standard supervised-learning assumptions (IID samples from ${\rho }_{{\pi }^{\ast }}$). The gap between sample and population error is the usual generalization gap.

Visual: draw a 1D axis of states labeled by ${\rho }_{{\pi }^{\ast }}$ (a bump centered on the expert's track). The training loss is the area under $\mathcal{L}(s){\rho }_{{\pi }^{\ast }}(s)$. Cloning squeezes that area to near zero, but says nothing about states where ${\rho }_{{\pi }^{\ast }}$ is near zero (off-track states).

17.5.3 Worked Examples

Additional quick check: if the dataset had 10 copies of $(s_2,-5)$ and the network averaged them, the optimum under squared error is the mean, $-5$, confirming that squared error pulls predictions toward the average expert action per state.

17.5.4 Student Questions and Answers

17.5.5 Industry Applications

Real-world: Behavior cloning is the baseline for many applied pipelines. A company collects human demonstrations for warehouse picking, trains a network with a simple squared-error loss on joint angles, and obtains a workable initial policy before adding more sophisticated correction. For example, collect 500 demos of a robot arm picking boxes (state = wrist camera view, action = 7-DOF joint targets), train with mean squared error, and get a policy that succeeds on typical box poses. Failures on unusual poses then motivate DAgger-style corrections.

17.5.6 Exam Notes

17.6 Why Naive Cloning Fails — Compounding Error and Distribution Mismatch

17.6.1 The Core Problem in Plain Words

Behavior cloning is simple and efficient, and it works when one-step deviation is not catastrophic and when expert trajectories cover the space well. If you make a small steering mistake but there are still nearby expert examples to pull you back, you stay safe. The trouble starts when the coverage is not that dense.

The key problem is a mismatch of data distribution — the states the learner sees at test time are not the same distribution it was trained on. Training saw states from ${\rho }_{{\pi }^{\ast }}$ (expert-visited). Testing with the learned policy generates states from ${\rho }_{\hat{\pi }}$ (learner-visited). After the first mistake those two differ, and the gap compounds.

17.6.2 The Red and Blue Track Story

Picture again the track with two lines. The blue line is the expert trajectory — where a good driver actually goes. The red line is the learner's path. Suppose the learner starts at the same spot as the expert and follows the blue line nicely for a while. At some point the supervised model suggests an action with a marginal deviation — a small steering error. That deviation alone is not serious; no one drives exactly on the center line.

What matters is what happens next. After that small shift, the car is now in a state that is slightly off the expert track. That exact off-track view may never have been encountered in any expert example. The network, which was only trained on on-track views, now has to guess. It often guesses poorly and produces an action that widens the gap. That new wider gap leads to an even more unfamiliar state, which leads to a larger mistake, and the error compounds.

The lecture walked this in steps: at the deviation point the red line is slightly off the blue line; feed that off-track state to the network; the expert may never have visited that precise state; the network gives some output; the error grows; soon the learner is in a region like a roadside strip it never saw during training but where it still assumes it can drive safely; the supervised network, learned only from the blue line, makes mistakes that pull it further away; eventually the car crashes. A small deviation leads to a new state, the new state leads to more mistakes, and the mistakes push the car further from the track.

This compounding is why the discussion notes that the network will actually try to give some output, so that error would slowly compound, and finally you will actually be crashing.

Visual: draw the oval track again. Blue line: smooth center loop. Red line: starts glued to blue, then a tiny kink outward at the top straight, then each segment bends further outward. Label the first kink "step $t$: $ϵ$-error → off-track state," the next segment "step $t+1$: state never seen → bigger error," and the final outward loop leaving the track "crash." Add two histograms below: left histogram ${\rho }_{{\pi }^{\ast }}$ tight around center; right histogram ${\rho }_{\hat{\pi }}$ shifted and spread outward.

17.6.3 Distribution Mismatch Formalized

During training the loss is averaged over ${\rho }_{{\pi }^{\ast }}$, the distribution of states the expert visits. At test time the learner generates its own distribution ${\rho }_{\hat{\pi }}$ by acting. Once it drifts, the sequence of states it collects — the "new distribution" drawn in red — no longer aligns with the expert's distribution. The learner keeps seeing states outside its training set and keeps making errors on them. This is the distribution mismatch problem.

In symbols, training assumes

$$s\sim {\rho }_{{\pi }^{\ast }}$$

but deployment sees

$$s\sim {\rho }_{\hat{\pi }}$$

and ${\rho }_{\hat{\pi }}\ne {\rho }_{{\pi }^{\ast }}$ after the first mistake. The further the learner drifts, the larger the gap between the two distributions, and the larger the average test error. The gap can be measured by total variation or by the extra loss $${\mathbb{E}}_{s\sim {\rho }_{\hat{\pi }}}[\mathcal{L}(\hat{\pi }(s),{\pi }^{\ast }(s))]-{\mathbb{E}}_{s\sim {\rho }_{{\pi }^{\ast }}}[\mathcal{L}(\hat{\pi }(s),{\pi }^{\ast }(s))]$$ which is zero only if the two distributions coincide.

A useful mental model: training loss is a flashlight that only illuminates the blue line; deployment walks with a flashlight that quickly points off the line into darkness.

17.6.4 Worked Examples

17.6.5 Student Questions and Answers

17.6.6 Industry Applications

Real-world: Highway lane-keeping with dense center-line data can succeed with pure cloning. Off-road or narrow track driving, where leaving the center means encountering unseen terrain, quickly reveals compounding error and forces a more interactive data strategy. For instance, a highway lane-keeping demo set with millions of centered frames tolerates small errors because the next frame still looks centered; a quarry road with one narrow path offers no such forgiveness — a 20 cm drift shows rocks never seen in training, the network mis-steers, and drift accelerates.

17.6.7 Exam Notes

17.7 Fixing the Mismatch with an Interactive Expert — DAgger

17.7.1 The Driving Instructor Analogy

The second major family fixes cloning by using an interactive demonstrator, much like learning to drive with an instructor beside you. At first you try to copy what the instructor says. With time you improve, but sometimes you deviate. When you do, the instructor takes control, adjusts the steering, and puts you back on track, helping you learn better. The instructor does not just give you a fixed log of examples; he intervenes when you make a mistake and gives you the correct action for the very state you just created.

That idea — let the learner act, and whenever it visits a new state, ask the expert what should have been done there — is the heart of the next algorithm.

17.7.2 DAgger in One Sentence

DAgger, short for Dataset Aggregation, alternates between collecting demonstrations, training a policy, letting the learner drive (partly on its own, partly with expert help), labeling the states the learner visited with the expert's correct actions, adding those new labeled states to the dataset, and retraining. The dataset grows so that it increasingly covers the states the learner actually encounters, not just the states the expert would have visited alone.

The lecture emphasized: collect demonstrations, learn from the expert via supervised learning, get a policy ready, go drive with it, let the interactive demonstrator make corrections, update the learner, and repeat. The loss itself remains the same simple gap between your action and the expert's action, often squared error, so the new idea is about what data you train on and how you mix control.

17.7.3 Mathematical Formulation

Let $\mathcal{D}$ be the aggregated dataset of state-action pairs, initially empty, written

$$\mathcal{D}=\varnothing$$

Let $\Pi$ be the policy class — all policies the network could represent — and let ${\hat{\pi }}_1\in \Pi$ be the first policy, initialized arbitrarily. In words, the instruction was to "initialize a policy Pi one to any policy; Pi is the space of all policies and Pi one is one policy within that space" which reconstructs to ${\hat{\pi }}_1$ chosen at random from $\Pi$.

DAgger runs in a loop over iterations $i=1,2,\dots ,N$. At iteration $i$ it uses a mixed policy ${\pi }_i$ that blends the expert ${\pi }^{\ast }$ and the current learner ${\hat{\pi }}_i$ via a parameter ${\beta }_i\in [0,1]$. The verbal description was "this beta is a parameter that instructor uses as to how much to intervene; when you begin this beta will be pretty high, meaning the policy would be largely by whatever your instructor gives; with time he will slowly decrease his involvement to ensure you become autonomous." That reconstructs to

$${\pi }_i(s)={\beta }_i{\pi }^{\ast }(s)+(1-{\beta }_i){\hat{\pi }}_i(s)$$

Interpreted as a stochastic mixture: at each step, follow ${\pi }^{\ast }$ with probability ${\beta }_i$ and follow ${\hat{\pi }}_i$ with probability $1-{\beta }_i$. In the lecture's words, "$\beta$ is $0.7$ so $70\%$ time follow your instructor and $30\%$ time follow yours — you still have two physical policies and then you choose between them randomly." A deterministic blend is also possible in some tasks, but the randomized mixture is the typical implementation noted.

A second form that makes the mixing explicit is

$$a\sim {\pi }_i(\cdot \mid s)=\{\begin{array}{ll}{\pi }^{\ast }(s)& \text{with probability }{\beta }_i\\ {\hat{\pi }}_i(s)& \text{with probability }1-{\beta }_i\end{array}$$

where $a$ is the action taken, $s$ is the current state, ${\pi }^{\ast }(s)$ is the expert's suggestion, ${\hat{\pi }}_i(s)$ is the learner's suggestion, and ${\beta }_i$ controls how often the expert is in charge.

Once ${\pi }_i$ is fixed, sample a trajectory using ${\pi }_i$. A trajectory means a sequence of states $s_1,s_2,\dots ,s_T$ generated by rolling out ${\pi }_i$ in the environment. Those states are still unlabeled for learning purposes, so for each visited state $s$ attach the expert's correct action ${\pi }^{\ast }(s)$. This creates a new batch

$${\mathcal{D}}_i=\{(s,{\pi }^{\ast }(s))\mid s\text{ visited by }{\pi }_i\}$$

Then aggregate

$$\mathcal{D}\leftarrow \mathcal{D}\cup {\mathcal{D}}_i$$

and train a new classifier or regressor on the enlarged $\mathcal{D}$ to obtain the next policy

$${\hat{\pi }}_{i+1}=\arg \underset{\pi \in \Pi }{\min }{\mathbb{E}}_{(s,a)\sim \mathcal{D}}[\mathcal{L}(\pi (s),a)]$$

With experience ${\beta }_i$ is decreased, so the learner becomes independent. Early on it relies mostly on the expert; later it drives more on its own and the dataset already contains many of the off-track corrections it will need.

Symbols: $\mathcal{D}$ is the growing dataset, ${\mathcal{D}}_i$ is the batch from iteration $i$, ${\beta }_i$ is the mixing weight at iteration $i$ in $[0,1]$, ${\pi }^{\ast }$ is the expert, ${\hat{\pi }}_i$ is the learner at iteration $i$, and ${\pi }_i$ is the mixed rollout policy.

17.7.4 The Algorithm Step by Step

1. Start with $\mathcal{D}=\varnothing$. Choose an initial ${\hat{\pi }}_1$ at random from $\Pi$.
2. For $i=1$ to $N$:

a. Form the mixed policy ${\pi }_i={\beta }_i{\pi }^{\ast }+(1-{\beta }_i){\hat{\pi }}_i$ and roll it out to get states visited under ${\pi }_i$. b. For each visited state $s$, query the expert for ${\pi }^{\ast }(s)$ and form pairs $(s,{\pi }^{\ast }(s))$ into ${\mathcal{D}}_i$. c. Aggregate $\mathcal{D}\leftarrow \mathcal{D}\cup {\mathcal{D}}_i$. d. Train a new policy ${\hat{\pi }}_{i+1}$ on $\mathcal{D}$ with the simple loss $\Vert \hat{\pi }(s)-{\pi }^{\ast }(s){\Vert }^2$. e. Reduce ${\beta }_{i+1}<{\beta }_i$ so the learner takes more control.

1. Return the best ${\hat{\pi }}_i$ on a held-out check or the final one.

Each step has a reason: the mixture creates states the learner would actually see, the expert label turns those states into supervised examples, aggregation remembers all past corrections so the learner does not forget, and decaying $\beta$ pushes autonomy.

Visual: a loop diagram with four boxes — "Roll out ${\pi }_i$" → "Label visited $s$ with ${\pi }^{\ast }(s)$" → "Aggregate $\mathcal{D}\cup {\mathcal{D}}_i$" → "Train ${\hat{\pi }}_{i+1}$" — with an outer arrow labeled "${\beta }_i\downarrow$" showing the instructor stepping back each round.

17.7.5 Worked Examples

17.7.6 Student Questions and Answers

17.7.7 Industry Applications

Real-world: DAgger-style correction is used when a safety driver can intervene. In data collection for autonomous driving, the car runs its current network, the human takes over at the edge of an error, and that takeover frame is logged as $(s,{\pi }^{\ast }(s))$ for the next training round. Similarly, in robotic grasping the robot attempts a grasp, a human corrects the wrist pose when it fails, and that corrected pose becomes a new $(s,a)$ pair on an otherwise rare failure state.

17.7.8 Exam Notes

17.8 Inverse Reinforcement Learning — Learning the Reward Behind the Behavior

17.8.1 Why "Inverse"

The phrase "inverse" can be confusing. In a typical, or forward, reinforcement learning setup you start with states, actions, often transition dynamics $P(s^{\prime }\mid s,a)$, and a reward function $R(s,a)$, and your goal is to learn an optimal policy ${\pi }^{\ast }$. In short

$$(\mathcal{S},\mathcal{A},P,R)⟶{\pi }^{\ast }$$

So you assume access to the world model and the reward, and you compute how to act.

Inverse reinforcement learning flips that. You are given states, actions, transitions, and many samples — demonstrations — but you do not know the reward. Your target is to learn the reward function itself. In words from the session: "your target is to learn the reward function; you will be given states and actions, you will be given transitions, and many samples, so you understand why it is coming under imitation learning — when it comes to imitation, you try to learn the policy from demonstrations, but here the primary objective is to learn the reward function, not the policy directly."

Once you recover a good reward, you can then use standard reinforcement learning to obtain a policy from it. So the chain becomes

$$\text{demonstrations }+(\mathcal{S},\mathcal{A},P)⟶\hat{R}⟶\hat{\pi }$$

where $\hat{R}$ is the learned reward and $\hat{\pi }$ is the policy that optimizes it. The arrow $\hat{R}\to \hat{\pi }$ is just ordinary RL with the inferred reward.

Visual: two opposite arrows. Top row: box with $(P,R)$ plus an arrow to ${\pi }^{\ast }$. Bottom row: box with demos plus $(P)$ plus a backward curved arrow labeled "inverse" pointing to $\hat{R}$, then a forward arrow to $\hat{\pi }$.

17.8.2 The Intuition: What Is the Expert Trying to Maximize

The motivation is that an expert's behavior is governed by some internal reward. Ask: why does the person do what they do? What would they gain? They are maximizing some reward function, even if they never write it down. If you can learn that reward, you understand their intent, not just their surface actions.

The lecture put it as "there is a reward function that he internally uses that governs his behavior, so what you try to learn is what he tries to maximize or minimize." In certain problems this is called reward modeling. If you crack the reward, you can reuse it to learn a policy that generalizes to new settings where the expert never demonstrated, because you now know what to optimize.

This is also the lens for interaction with language models. When you chat with a model, the model tries to understand what kind of response gives you higher satisfaction — what your internal reward is. It builds a picture of your preferences and then produces outputs that score well under that model of your reward. That problem is fundamentally a reward-modeling problem, and it draws on inverse reinforcement learning ideas.

The key property of inverse RL is transfer: the learned $\hat{R}$ lets you act well where no demo exists, because you optimize $\hat{R}$ there. Pure cloning has no such transfer.

17.8.3 Mathematical Formulation

You still model the reward with a parametric form that takes state and action as input and predicts a scalar score. The session illustrated the linear case first, saying "assume you are extracting many features from state and action — feature one, feature two, feature three, feature four — and you are trying to learn the weight of each feature."

Let $\varphi (s,a)=[F_1(s,a),F_2(s,a),F_3(s,a),F_4(s,a){]}^{\top }$ be a feature vector extracted from $(s,a)$, where each $F_j(s,a)\in \mathbb{R}$ is a hand-designed or learned feature such as distance to a wall or speed, and let $\theta =[{\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4{]}^{\top }\in {\mathbb{R}}^4$ be learnable weights plus a bias ${\theta }_0\in \mathbb{R}$. Then the linear reward model is

$$R_{\theta }(s,a)={\theta }_1{F}_1(s,a)+{\theta }_2{F}_2(s,a)+{\theta }_3{F}_3(s,a)+{\theta }_4{F}_4(s,a)+{\theta }_0$$

In vector form

$$R_{\theta }(s,a)={\theta }^{\top }\varphi (s,a)+{\theta }_0$$

where $R_{\theta }(s,a)\in \mathbb{R}$ is the predicted reward at $(s,a)$, ${\theta }^{\top }$ is the transpose of the weight vector, and $\varphi (s,a)\in {\mathbb{R}}^4$ is the feature column. A nonlinear model replaces the dot product with a network, written $R_{\theta }(s,a)=f_{\theta }(s,a)$, where $f_{\theta }$ is a neural net with parameters $\theta$. The learning goal is to choose $\theta$ so that the observed demonstrations look near-optimal under $R_{\theta }$ — the reward that best justifies the behavior you saw.

How is $\theta$ chosen? Conceptually, pick $\theta$ so that expert trajectories score higher than alternative trajectories under $R_{\theta }$. Different algorithms make this precise by maximizing the gap between expert return and non-expert return, or by matching feature expectations ${\mathbb{E}}_{{\pi }^{\ast }}[\varphi ]$ to ${\mathbb{E}}_{\hat{\pi }}[\varphi ]$. The session kept scope conceptual: know what inverse reinforcement learning is and how it differs from forward reinforcement learning, rather than reproducing a full solver.

For the linear model the return of a trajectory $\tau$ is

$$J_{\theta }(\tau )=\sum _t{R}_{\theta }(s_t,a_t)={\theta }^{\top }\left(\sum _t\varphi (s_t,a_t)\right)+T{\theta }_0$$

so learning $\theta$ is learning how to weight summed features along a trajectory. The bias ${\theta }_0$ sets a baseline offset and does not affect comparison between trajectories of equal length.

Visual: picture a table with four feature columns $F_1$ to $F_4$ and a weight column ${\theta }_1$ to ${\theta }_4$. Each row is a state-action pair; the reward is the weighted sum across the row. Learning tilts the weight column so that rows from expert trajectories sum high and other rows sum low.

17.8.4 Worked Examples

17.8.5 Student Questions and Answers

17.8.6 Industry Applications

Real-world: In game analysis, ghosting plus inverse reinforcement learning estimates a player's hidden preferences — for example, how a striker weights a quick shot versus a safer pass — which feeds coaching and opposing-team planning. In language models, reward models are trained from human feedback and then used to fine-tune the assistant to fit what users actually find helpful. A reward model $R_{\theta }(s,a)$ where $s$ is a prompt and $a$ is a response learns to score responses by human preference; the assistant is then optimized to produce high-$R_{\theta }$ responses. Both are direct applied uses of inverse reinforcement learning ideas where the reward is about user satisfaction and helpfulness.

17.8.7 Exam Notes

17.9 A Small Concrete Illustration — The Recycling Robot and Poor Reward Design

17.9.1 The Setup

A compact example that was shown uses a recycling robot with two battery states, $s=\text{high}$ and $s=\text{low}$, and two actions, $a=\text{search}$ and $a=\text{recharge}$. The expert's preferred behavior is simple to state: when the battery is high, search; when the battery is low, recharge. The learner's initial attempt does the opposite in one state — it searches when it should recharge or vice versa — so there is a visible mismatch.

Let ${\pi }^{\ast }(\text{high})=\text{search}$ and ${\pi }^{\ast }(\text{low})=\text{recharge}$ be the expert's choices, and let $\hat{\pi }(\text{high})=\text{search}$ but $\hat{\pi }(\text{low})=\text{search}$ be a learner that is lazy about recharging. On state "high" the pair matches, on state "low" it does not. The training loss counts that misalignment as $1$ for the mismatched state and $0$ for the matched one, so the gap is visible and can be improved by adding demonstrator examples.

17.9.2 Reward Misspecification in Numbers

Now change the lens to reward. Suppose the world gives the agent $0.5$ reward each time step it waits, and $1$ each time it searches while battery is high, but also some cost if it searches while low. If the reward model is poorly learned and the agent believes waiting gives $0.5$ for free with no risk, it may reason: why sweat to search? Sit back and collect $0.5$ every minute — over many steps that sums to a large total. That learned reward makes the lazy policy look optimal when it is not. The point of the example is that small mistakes in reward modeling change the optimal behavior completely. A correct reward makes searching when able the best choice; a weak reward model makes idling look attractive.

Concretely, over $100$ steps, waiting yields $100\times 0.5=50$ under the flawed model, while correctly search-and-recharge might yield a higher true return but is never discovered because the flawed reward never encourages it. The lecture flagged this as the kind of trouble poor reward modeling causes, not as a lengthy calculation to memorize.

Visual: draw two timelines of 10 steps. Top timeline: "wait" every step, reward $0.5$ each, cumulative $5$ with flat line. Bottom timeline: "search when high, recharge when low" with rewards $1,1,0,1,1,0\dots$ cumulative $6$ and rising faster. Label the flawed model's view where the flat line is drawn as if it ends higher.

17.9.3 Worked Examples

17.9.4 Industry Applications

Real-world: In warehouse robots, a reward that only counts short-term pick count can produce a policy that never charges, leading to mid-shift failures. Correct reward design adds a term for battery health, learned from expert demonstrations of when a good operator chooses to charge. Concretely, if the dispatcher's demos show charging at 30% battery even though picks are pending, an inverse-RL reward that includes $F_{\text{battery}}$ will learn a negative weight for "search while low," reproducing the expert's charging behavior and avoiding the lazy wait-like failure.

17.9.5 Exam Notes

Exam Guidance Summary

This session contained several explicit signals about what matters for assessment and what does not:

• The numerical depth on TPO loss functions covered previously is not in scope for this course and was skipped for exam preparation.
• Imitation learning itself is in scope. Be able to define it, explain expert demonstrations as state-action pairs, and describe the supervised reduction.
• Behavior cloning: know the objective "minimize expected loss on the expert's state distribution," write $\hat{\theta }=\arg {\min }_{\theta }{\mathbb{E}}_{s\sim {\rho }_{{\pi }^{\ast }}}[\mathcal{L}({\hat{\pi }}_{\theta }(s),{\pi }^{\ast }(s))]$ with definitions of each symbol, and write the squared-error loss $\Vert {\hat{\pi }}_{\theta }(s)-{\pi }^{\ast }(s){\Vert }^2$. Know why it is simple and when it works. Expect conceptual questions, not long numerical drills.
• Distribution mismatch and compounding error: be able to narrate the blue-expert versus red-learner drift, define ${\rho }_{{\pi }^{\ast }}$ versus ${\rho }_{\hat{\pi }}$, and explain "small deviation to unseen state to larger error to crash." Be ready to write $s\sim {\rho }_{{\pi }^{\ast }}$ for training and $s\sim {\rho }_{\hat{\pi }}$ for deployment and note ${\rho }_{\hat{\pi }}\ne {\rho }_{{\pi }^{\ast }}$.
• DAgger: memorize and reproduce the full loop — $\mathcal{D}=\varnothing$, random ${\hat{\pi }}_1$, mixture ${\pi }_i={\beta }_i{\pi }^{\ast }+(1-{\beta }_i){\hat{\pi }}_i$ with ${\beta }_i$ decay, rollout of ${\pi }_i$, expert relabeling to form ${\mathcal{D}}_i$, aggregation $\mathcal{D}\leftarrow \mathcal{D}\cup {\mathcal{D}}_i$, retraining. Know the stochastic form with probability ${\beta }_i$. Explain why each line helps. No heavy numerical is expected; the slides attributed to a teaching assistant contain numbers for reference only and are not the focus.
• Inverse reinforcement learning: contrast forward $(S,A,P,R)\to \pi$ with inverse $(\text{demos},S,A,P)\to R\to \pi$. Write the linear reward $R_{\theta }(s,a)={\theta }^{\top }\varphi (s,a)+{\theta }_0$ and explain that it learns the intent behind behavior. Be ready to give a real-life example such as player intent in soccer or reward modeling for interactive language models that learn user satisfaction.
• Recycling robot: treat it as intuition for behavior alignment and for "poor reward gives poor policy, e.g., waiting at $0.5$ looks better than searching at $1$ when the model is wrong," not as a calculation drill. Know the high/low table and the mismatch count.
• Optional topics listed in the shared notes should be ignored. Study from the instructor's notes using the guidance from the class, paying attention to the motivation with which the notes were written in elaborate form.

Real-world exam advice that was shared: write all assumptions explicitly, present work in a table when the problem is tabular so grading is easier, and focus revision on the concepts emphasized live rather than on optional numerical appendices.

Key Industry Applications

• Autonomous driving lane-keeping (NAVLAB pattern): Camera view plus human steering logged as $(s,a)$ pairs; a network maps view to wheel angle; behavior cloning provides the baseline, DAgger-style human takeovers enrich the dataset with recovery states. Carnegie Mellon NAVLAB showed this works even with a simple network on a confined road.
• Game ghosting and strategy: Soccer and basketball trajectories from many past games create ghost overlays of "what an average or expert player would do here"; deviation detection and learned trajectory models feed coaching, opponent modeling, and play design; the same multi-trajectory learning extends to any setting with many recorded human plays. Ghost trajectories superimposed on live play make the expected policy visible.
• Helicopter acrobatics and robotics: Difficult maneuvers that are hard to encode as a reward are learned from demonstrations, as shown in Stanford helicopter work co-authored by Andrew Ng; similarly, surgical or assembly robots copy skilled operator traces. The demo trace carries timing that manual reward writing misses.
• Reward modeling for interactive systems: Language-model interaction is framed as learning the user's hidden reward — what kind of answer the user values — so that the system can produce responses that maximize that inferred reward; this is a direct applied use of inverse reinforcement learning ideas, where the learned $R_{\theta }(s,a)$ scores prompt-response pairs by satisfaction and helpfulness.
• Warehouse and field robots: The recycling-robot intuition generalizes to battery-aware policies; a misspecified reward that rewards idling at $0.5$ can make a lazy wait-loop look optimal, so learning a reward that reflects true expert trade-offs — for example, weighting battery health alongside pick count — is essential for long-horizon performance.