explainer
DAgger imitation learning: label the states your policy visits
Run DAgger imitation learning one round at a time. Collect expert labels on learner rollouts, retrain on the growing dataset, and compare equal label budgets.
What you will learn
- Trace one round of learner rollout, expert labeling, aggregation, and retraining.
- Distinguish the executed learner action from the expert training label.
- Compare collection methods at the same expert-query budget.
- Keep evaluation episodes separate from fitting and checkpoint selection.
- Identify coverage and policy limits that dataset aggregation leaves unresolved.
Before you start
A learner can visit states its expert demonstrations barely cover. DAgger asks the expert what to do at those visited states. It adds those answers to the training set, then fits the next policy.
This lesson runs that loop. You will inspect each batch and compare two collection methods with equal expert-label budgets. The first round also shows a useful warning: more relevant data can still worsen a finite task score.
What DAgger changes about imitation learning
DAgger, short for Dataset Aggregation, changes how an imitation learner collects training inputs. Its current policy visits states, and an expert supplies action labels there. Training keeps the accumulated labeled data across rounds.
Ordinary behavior cloning fits a policy to demonstrations. The MIT imitation-learning notes describe this supervised policy-learning problem. The expert determines the training states when it also controls the demonstrations.
A learned policy can produce different state sequences. This covariate shift changes which predictions matter during execution. Ross, Gordon, and Bagnell introduced DAgger to address learning under the policy's own state distribution. Original DAgger paper
Follow the dataset aggregation loop
Start with expert demonstrations and fit an initial policy. Each additional round follows four steps:
- Freeze the current policy and run collection episodes.
- Ask the expert to label every visited pre-action state.
- Append those state-label pairs to the existing dataset.
- Fit the next policy using the full dataset.
Dᵢ₊₁ = Dᵢ ⊎ [(x, π*(x)) for x visited by πᵢ]
Here ⊎ means concatenation, including repeated queries. The expert's answer becomes a training target. The learner's executed action determines the next state during collection.
The original algorithm can mix expert and learner actions using a probability β. Our experiment uses initial expert demonstrations, then learner-only collection with β = 0. This follows the paper's simplest collection pattern. Algorithm 3.1
Define the robot, expert, and learner
A point robot moves along a line toward zero. Its observation equals its true position x in meters. Its action u is an exact displacement in meters:
x[k+1] = x[k] + u[k]
π*(x) = −x
The expert reaches zero in one update and then stays there. The model has no obstacles, sensing noise, velocity limits, or travel duration. These assumptions isolate data collection from physical control and actuation.
The learner uses one nearest neighbor. It copies the expert label stored at the closest training position. Distances tied within 10⁻¹² m choose the lower stored position, which makes every replay deterministic.
Fitting this learner means retaining the labeled examples for nearest-neighbor lookup. It performs no interpolation or gradient optimization. A linear learner could recover u = −x from these simple data; our restricted learner exposes the effect of collection coverage.
The shared initial dataset contains three separate one-action expert episodes: x = −0.1, 0, and 0.1 m. Both methods begin with those same three labels.
Separate collection from evaluation
Each collection round runs two five-update episodes. The schedule fixes their starting positions before either method trains:
| Round | Both sides: starts (m) | Positive starts: starts (m) |
|---|---|---|
| 1 | −0.83, 0.83 | 0.83, 0.63 |
| 2 | −0.62, 0.62 | 0.62, 0.42 |
| 3 | −0.91, 0.91 | 0.91, 0.71 |
| 4 | −0.74, 0.74 | 0.74, 0.54 |
DAgger collection executes the frozen learner and requests expert labels at all ten visited states. Expert-only collection executes the expert from the same two starts and records ten labels. Both methods append the batch and refit the same learner type. The imitation-learning methods map separates this data-collection choice from the policy architecture and training objective.
The fixed evaluation starts are −0.78, −0.58, −0.38, −0.04, 0.04, 0.38, 0.58, and 0.78 m. Every policy gets five updates from each start. We report mean final distance from zero and count successes with |x[5]| ≤ 0.025 m.
Collection and fitting never receive evaluation episodes or scores. The round selector shows a requested checkpoint; the program never chooses a winning checkpoint from evaluation results. A training rollout can naturally revisit a numeric state that another episode also reaches.
This fixed suite is an authored teaching example, not an untouched research benchmark. Changing design choices after inspecting its scores would require a fresh test set for a generalization claim. The robot data leakage lesson explains that separation.
Calculate the first two rounds
Start round 1 at x = 0.83 m. The initial learner's closest demonstration is 0.1 m, so it executes u = −0.1 m. Its five queried positions are:
0.83 → 0.73 → 0.63 → 0.53 → 0.43
At the first state, the expert labels u = −0.83 m*. That differs from the learner's executed −0.1 m. The fifth action reaches 0.33 m; this terminal state receives no label in this five-query episode.
The negative episode supplies the opposite positions and labels. After aggregation, both methods hold 3 + 2 × 5 = 13 expert labels. Expert-only collection has positions ±0.83 plus eight repeated zeros in its new batch.
In round 2, the frozen DAgger policy starts at 0.62 m. Its closest stored state is 0.63 m, so it commands −0.63 m and reaches −0.01 m. Zero is now its closest example, so it stays at −0.01 for the remaining collection steps.
The expert labels every visit to −0.01 with +0.01 m. The other episode supplies the opposite correction. Retraining on all 23 labels therefore adds small recovery actions near the goal.
At evaluation start 0.78 m, the round-2 learner follows 0.78, 0.05, 0.04, 0.03, 0.02, 0.01 m. The first tie chooses stored x = 0.73 m over 0.83 m. Subsequent actions use the new −0.01 m label at stored x = 0.01 m.
Inspect the growing dataset and rollout
Begin with Both sides and round 0. Advance to round 1, then round 2. The coverage rows show which states generated labels; the rollout shows what the newly fitted policy does afterward.
Open the collected-label table to compare each executed action with its expert target. Repeated marks overlap in the coverage plot. The label count includes every query, including repeated states.
Changing the evaluation start changes the displayed episode. The aggregate scores still use all eight fixed starts. Changing collection starts preserves the selected round and evaluation start.
Read improvement and failure together
With Both sides selected, the comparison is:
| Round | Labels per method | DAgger mean final error (m) | Expert-only mean final error (m) | Successes: DAgger / expert-only |
|---|---|---|---|---|
| 0 | 3 | 0.105000 | 0.105000 | 2 of 8 / 2 of 8 |
| 1 | 13 | 0.047500 | 0.040000 | 0 of 8 / 2 of 8 |
| 2 | 23 | 0.005000 | 0.042500 | 8 of 8 / 0 of 8 |
| 3 | 33 | 0.000000 | 0.042500 | 8 of 8 / 0 of 8 |
| 4 | 43 | 0.000000 | 0.040000 | 8 of 8 / 0 of 8 |
Displayed zeros round to six decimal places. Round 1 reduces DAgger's average distance while its success count falls. New nearest neighbors change the action regions, and some previously successful starts finish outside the chosen tolerance.
Now select Positive starts, round 4, and evaluation start −0.78 m. DAgger ends at −0.28 m, missing the goal. Its total success count is 6 of 8, with mean final error 0.046250 m.
That collection schedule never supplies the missing large negative-state examples. Small excursions below zero supply some recovery labels, but they leave this coverage gap. The example shows missed goals and slow recovery; it establishes no general divergence claim.
For a larger evaluation, predeclare episode conditions, stop rules, and metric denominators. The policy-evaluation experiment shows those choices and explains when a success-rate interval is justified.
Count labels and collection costs honestly
Both methods spend ten additional expert queries per round. They do not collect equal numbers of distinct states. The expert-only episodes reach zero immediately, so eight of each batch's ten queries repeat the goal state at zero.
We keep those queries and disclose the redundancy. A collector that stopped completed episodes or bought more fresh starts would make a different comparison. Equal query counts also leave human attention, reset time, and robot wear unmeasured.
All stored targets here come from a perfect analytic expert. A human teacher can face ambiguous observations, delayed feedback, or unsafe recovery states. Any robot experiment needs its own collection constraints and intervention record.
Understand the scope of DAgger guarantees
The original analysis connects DAgger with no-regret online learning. Its results depend on assumptions about the learner, loss, sampling, and expert mixing; task-cost bounds also require a suitable relation to imitation loss. They do not promise that every successive policy improves. Ross, Gordon, and Bagnell, Sections 3–4
Our nearest-neighbor experiment has no established no-regret guarantee. Its eight authored episodes show specific outcomes under exact discrete dynamics. They provide no hardware-transfer or safety evidence.
DAgger also needs useful expert answers at encountered states. Aggregation alone cannot fix missing observations, inconsistent labels, an inadequate policy class, or an unreachable task. Start with a precise data-collection and evaluation question for the robot you actually have.
Reproduce the rounds in Python
This standard-library script stores labeled positions; the expert label is always their negative. Each round freezes the old dataset until both collection episodes finish. Evaluation occurs after training and never feeds back into it.
EVAL = (-.78, -.58, -.38, -.04, .04, .38, .58, .78)
MAGNITUDES = (.83, .62, .91, .74)
def action(data, x):
nearest = data[0]
for sample in data[1:]:
delta = abs(sample - x) - abs(nearest - x)
if delta < -1e-12 or (abs(delta) <= 1e-12 and sample < nearest):
nearest = sample
return -nearest
def rollout(data, x):
points = [x]
for _ in range(5):
x += -x if data is None else action(data, x)
points.append(x)
return points
def train(positive=False):
dagger, baseline = [-.1, 0., .1], [-.1, 0., .1]
history = [(dagger, baseline)]
for magnitude in MAGNITUDES:
starts = (magnitude, magnitude - .2) if positive else (-magnitude, magnitude)
labels = [x for start in starts for x in rollout(dagger, start)[:-1]]
expert_labels = [x for start in starts for x in rollout(None, start)[:-1]]
dagger = dagger + labels
baseline = baseline + expert_labels
history.append((dagger, baseline))
return history
def score(data):
errors = [abs(rollout(data, start)[-1]) for start in EVAL]
return sum(errors) / len(errors), sum(error <= .025 + 1e-12 for error in errors)
def report(name, pair):
dagger, baseline = pair
dm, ds = score(dagger)
bm, bs = score(baseline)
print(f"{name}: labels={len(dagger)} DAgger={dm:.6f} ({ds}/8) expert-only={bm:.6f} ({bs}/8)")
history = train()
for index, pair in enumerate(history):
report(f"round {index}", pair)
report("positive round 4", train(positive=True)[4])
print("round 2 x:", ", ".join(f"{x:.6f}" for x in rollout(history[2][0], .78)))
Output:
round 0: labels=3 DAgger=0.105000 (2/8) expert-only=0.105000 (2/8)
round 1: labels=13 DAgger=0.047500 (0/8) expert-only=0.040000 (2/8)
round 2: labels=23 DAgger=0.005000 (8/8) expert-only=0.042500 (0/8)
round 3: labels=33 DAgger=0.000000 (8/8) expert-only=0.042500 (0/8)
round 4: labels=43 DAgger=0.000000 (8/8) expert-only=0.040000 (0/8)
positive round 4: labels=43 DAgger=0.046250 (6/8) expert-only=0.072500 (1/8)
round 2 x: 0.780000, 0.050000, 0.040000, 0.030000, 0.020000, 0.010000
Try it yourself
Exercise 1. At the first collection state x = −0.83 m, what action does the initial learner execute? What expert label enters the dataset, and what state does the learner visit next?
Show solution: separate the action from its label
The closest demonstration is x = −0.1 m, whose label is +0.1 m. The learner executes that displacement and visits −0.73 m. The expert supplies +0.83 m as the training label at the original state.
Executing the expert label would reach zero immediately. DAgger collection here keeps the learner in control, so the next query addresses its own visited state.
Exercise 2. After round 2 with Both sides, the absolute final errors are 0.01, 0, 0.01, 0, 0, 0.01, 0, and 0.01 m, up to floating-point roundoff. Calculate mean final error, success count, and expert labels per method.
Show solution: compare error, success, and budget
The errors sum to 0.04 m. Dividing by eight gives 0.005 m mean final error. Every error is at most 0.025 m, so all 8 of 8 episodes succeed.
Each method starts with three labels and adds ten in each of two rounds: 3 + 2 × 10 = 23 labels. This count includes duplicate queries; it measures neither unique coverage nor human collection time.
Sources and further study
- Ross, Gordon, and Bagnell: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. The 2011 paper introduces DAgger; its PDF gives Algorithm 3.1 and the assumptions behind the analysis.
- Russ Tedrake: Imitation Learning, Underactuated Robotics. The state-action regression formulation supplies the supervised-learning context for this experiment.
Continue with robot data leakage to design episode splits and keep evaluation results from shaping your reported test set.