Imitation learning: compare methods by data and expert access

Imitation learning uses demonstrated behavior to learn how to act. Some methods need action labels. Others infer an objective or compare the behavior a policy produces.

The useful first question is what data and access you have. Can the expert label new states? Can the learner collect more rollouts, or must training use a fixed recording?

This guide compares four workflows and a missing-action setting. You will calculate an action loss and see why one expert choice can support several costs. The diagrams describe methods; they report no robot-training results.

For a sequence of small experiments after this comparison, follow a policy from demonstrations to evaluation.

Define imitation learning through its evidence

Imitation learning, also called learning from demonstration, uses another agent's behavior as learning evidence. A demonstrator might provide joystick commands, joint motions, or recorded task trajectories. Calling that demonstrator an expert expresses a modeling assumption about the usefulness of its behavior.

Use s for a physical state, o for an observation, and a for an action. A policy chooses actions from available information, possibly including observation history. A trajectory records a sequence through time.

MIT's imitation-learning notes distinguish state-action demonstrations from observation-action sequences. That distinction matters when sensors hide part of the state. A learner cannot use information that its deployed observation interface never provides.

Compare the data and access requirements

Compare data and training access

Four ways to use expert behavior

Read each panel downward. Blue return arrows repeat collection and training.

Behavior cloning

DAgger collection

Inverse reinforcement learning

GAIL

These panels show representative workflows. They separate data, objectives, and access requirements; methods and policy architectures can combine.

These are comparison axes, not mutually exclusive categories. Behavior cloning describes a supervised policy-fitting approach; DAgger describes an interactive collection and refitting procedure. A larger system can combine several of these ideas.

Workflow shown	Expert supplies	Additional training access
Fixed-data behavior cloning	Observation-action examples	No new rollout or expert query required for fitting
DAgger	Initial demonstrations and labels on visited states	Learner collection episodes and an expert who can answer new queries
Inverse RL, then policy optimization	Behavior that constrains a reward or cost	A dynamics model or environment samples, according to the chosen algorithm
Original GAIL	Recorded state-action trajectories	New learner rollouts; no new expert action queries required

These rows describe training access. Every resulting robot policy still needs task evaluation. A dataset that permits fitting does not establish deployment readiness.

Fit demonstrated actions with behavior cloning

Behavior cloning (BC) uses supervised learning to fit demonstrated actions. For continuous actions, a deterministic policy can minimize squared prediction error. Other policies learn conditional action distributions with objectives suited to those distributions.

Here θ represents learned parameters and the star marks the expert label. This is a regression example, not a definition that restricts all BC to squared error. Tedrake's state-feedback formulation provides the supervised-learning connection.

A fixed dataset can support fitting without running the learner during training. During execution, its actions affect its next observations. The behavior-cloning lesson compares prediction error with complete rollouts, and the covariate-shift lesson examines changing state distributions.

Ask for labels on learner states with DAgger

DAgger, or Dataset Aggregation, adds expert labels at states the collection policy visits. It retains earlier labels and refits on the growing dataset. Ross, Gordon, and Bagnell's Algorithm 3.1 specifies that loop and allows expert-policy mixing during collection.

The expert's label answers what it would do at a visited state. That answer can differ from the learner's executed action. The displayed map shows learner-only collection after initial demonstrations; mixed collection is another option.

This requires access to an expert who can label new situations, plus a way to run collection episodes. The DAgger experiment performs actual aggregation and compares equal expert-query budgets. Its successive policies also show why a chosen task score need not improve every round.

Infer an objective with inverse reinforcement learning

Inverse reinforcement learning (IRL) seeks a reward or cost that explains observed behavior under stated assumptions. A reward-maximizing and cost-minimizing description can express the same task by setting reward equal to negative cost. Producing actions then requires planning or policy learning with that objective.

Ng and Russell's 2000 paper studies this inverse problem in Markov decision processes. Its formulations include a known expert policy and a finite set of observed trajectories. The author's paper summary emphasizes that many rewards can explain the same policy.

The chosen IRL algorithm determines its dynamics assumptions and access needs. Model-based planning can use a transition model; sample-based approaches can use environment interaction. Recovering a plausible objective alone does not establish that it captures the demonstrator's unique intent.

Match visited behavior with GAIL

Generative Adversarial Imitation Learning (GAIL) trains a policy using expert and learner state-action samples. A discriminator distinguishes their sources, and its signal guides policy updates. Original GAIL collects new learner rollouts throughout this loop, while reusing recorded expert trajectories. Ho and Ermon, Algorithm 1

Its occupancy-matching view compares which state-action pairs policies visit and how much weight those visits receive over time. The paper relates this objective to regularized IRL followed by reinforcement learning. Its practical algorithm learns a policy without requiring a separate explicit reward-recovery stage.

The discriminator supplies a useful training signal. Reading that signal as a uniquely recovered task reward requires further assumptions. Finite data and optimization also limit what an empirical match establishes.

Account for missing action labels

Imitation from observation addresses demonstrations without the expert's action labels. A video can show motion while leaving actuator commands hidden. This is a data-availability distinction that can cut across method families.

Torabi, Warnell, and Stone's Behavioral Cloning from Observation uses the learner's own interaction data to learn an inverse dynamics model. That model infers actions from demonstrated state transitions, which then support policy fitting. The behavior-cloning-from-observation experiment performs both fits and shows how weak action diversity or measurement error can corrupt the inferred labels. Access to video alone does not supply the same information as labeled robot trajectories.

Consider the known scalar model x′ = x + a. Exact consecutive positions identify a = x′ − x. If the model instead has x′ = x + a₁ + a₂, the same observed displacement can come from many action pairs.

Missing actions therefore require assumptions about dynamics, observations, and the relation between demonstrator and learner. Camera viewpoint and embodiment differences add their own inference problems. State-transition matching also needs a representation that makes the comparison meaningful.

Separate methods, architectures, and RL

A neural network is a policy representation. A diffusion policy represents conditional action sequences through a denoising model. A vision-language-action model conditions actions on visual and language inputs, with a design-specific decoder. The robot foundation-model overview compares pretraining data, adaptation, and transfer evidence across published policies; the foundation-model label alone does not identify a training objective.

Supervised learning fits input-target pairs; BC applies that idea to action prediction. DAgger keeps supervised fitting while changing which labeled pairs it collects.

Architecture names describe different choices from how demonstrations are collected. A diffusion policy can train on demonstrations through behavior cloning. An architecture name alone does not tell you whether training later adds expert feedback or reward-based updates.

Reinforcement learning optimizes behavior using a reward or cost signal. In imitation workflows, an inferred objective or discriminator can supply that signal, and RL can perform policy optimization. Thus a training procedure can use demonstrations and RL together; its actual data and objective explain what it learns.

Calculate action loss and reward ambiguity

First, take three position observations in meters. The expert commands displacement a* = −x, and a candidate policy makes these predictions:

Position x (m)	Expert a* (m)	Predicted a (m)
−0.4	0.4	0.3
0	0	0.1
0.4	−0.4	−0.2

The squared errors are 0.01, 0.01, and 0.04 m². Their mean is 0.02 m². This calculation scores these three labeled actions; it specifies no rollout or task-success result.

Now consider a one-decision route choice. Give each route a distance d in meters and a dimensionless roughness score p. Define the toy cost cλ = d + λp, where λ has units of meters and lower cost is better.

Available route	Distance d (m)	Roughness p
A	2	1
B	3	0

Suppose the expert chooses B, and we assume it minimizes this cost family. That choice requires 3 m ≤ 2 m + λ, or λ ≥ 1 m. Strict preference for B requires λ greater than 1 m.

Both λ = 2 m and λ = 10 m explain a strict preference for B. They disagree when a new choice offers C = (1 m, 0.2) and D = (2 m, 0):

• At λ = 2 m, C costs 1.4 m and D costs 2 m. Choose C.
• At λ = 10 m, C costs 3 m and D costs 2 m. Choose D.

One demonstrated choice leaves an entire range of costs consistent with the observation. More varied evidence or additional assumptions can constrain that range. Roughness here is a supplied toy feature; these numbers establish no physical comfort or safety model.

This standard-library Python reproduces both calculations. It evaluates supplied predictions and candidate costs; it implements neither IRL nor GAIL.

expert = (0.4, 0.0, -0.4)
predicted = (0.3, 0.1, -0.2)
mse = sum((a - target) ** 2 for a, target in zip(predicted, expert)) / len(expert)
print(f"Action MSE: {mse:.6f} m^2")


def costs(routes, weight):
    return {name: distance + weight * roughness
            for name, (distance, roughness) in routes.items()}


def describe(values):
    return ", ".join(f"{name}={value:.1f}" for name, value in values.items())


demonstrated = {"A": (2.0, 1.0), "B": (3.0, 0.0)}
new_choice = {"C": (1.0, 0.2), "D": (2.0, 0.0)}
for weight in (2.0, 10.0):
    old = costs(demonstrated, weight)
    new = costs(new_choice, weight)
    print(f"lambda={weight:.0f}: {describe(old)} -> {min(old, key=old.get)}")
    print(f"lambda={weight:.0f}: {describe(new)} -> {min(new, key=new.get)}")

Output:

Action MSE: 0.020000 m^2
lambda=2: A=4.0, B=3.0 -> B
lambda=2: C=1.4, D=2.0 -> C
lambda=10: A=12.0, B=3.0 -> B
lambda=10: C=3.0, D=2.0 -> D

Choose a comparison your data can support

Before choosing a method to investigate, record four facts:

• Labels: which observations, actions, timestamps, and units do the demonstrations contain?
• Expert access: can the teacher answer new action queries after recording?
• Environment access: can the learner collect rollouts, and what model or simulator is available?
• Objective: do you need an action policy, an interpretable cost hypothesis, or a behavioral match?

Labeled offline demonstrations support a BC fitting experiment. New expert answers make a DAgger collection experiment possible. Recorded demonstrations plus learner rollouts support investigating original GAIL; an IRL study must also state its reward class and dynamics assumptions.

These conditions establish what you can test, not which method will win. Compare demonstration volume, expert queries, and environment interaction budgets explicitly. Keep test episodes out of fitting and model selection, as the data leakage lesson illustrates.

Report prediction loss alongside independent task outcomes. A matched demonstration score can hide failures under changed starts or observations. The robot policy evaluation lesson develops that reporting question.

Try it yourself

Exercise 1. Replace the candidate actions in the worked BC example with 0.4, 0, and −0.3 m. Calculate the new MSE. Does the lower loss establish that the robot will complete a task?

Show solution: score the labeled predictions

The squared errors are 0, 0, and 0.01 m². Their mean is 0.003333… m², which is lower than 0.02 m².

This comparison covers three supplied observation-action pairs. A task claim still needs defined dynamics, starting conditions, and rollouts with a success criterion. The calculation supplies none of those outcomes.

Exercise 2. For the route-cost family, find every λ that makes demonstrated route B at least as good as A. Then find where the new routes C and D tie. Can the demonstrated choice alone determine the preferred new route?

Show solution: keep every consistent cost

B costs 3 m and A costs 2 m + λ, so λ ≥ 1 m makes B optimal among these two choices. At λ = 1 m, both routes tie.

C and D tie when 1 + 0.2λ = 2, giving λ = 5 m. Consistent values from 1 m to below 5 m prefer C; at 5 m they tie. Values above 5 m prefer D.

The single demonstration therefore leaves the new choice unresolved.

Follow a policy from demonstrations to evaluation

Start here if you can identify an observation–action pair and calculate an average squared error. The examples supply synthetic data and small robot models, so you can inspect the learning and evaluation choices without collecting demonstrations first.

1. How does a demonstration become an action rule? Fit a nearest-neighbor policy in behavior cloning, then compare held-out action error with what happens during its rollout.
2. Which examples should stay out of training? Use robot data leakage and episode splits to compare framewise and episodewise splits, inspect membership, and record a split manifest.
3. Why does the learner reach unfamiliar states? Follow covariate shift in behavior cloning to separate prediction error on expert states from errors during a learner's own rollout.
4. What changes if the expert can label those states? Run the DAgger experiment to collect states, request labels, aggregate data, and refit under an explicit expert-query budget.
5. What counts as a successful policy? Work through robot policy evaluation to declare success and stopping rules, compare matched episodes, and report failures alongside success rates.

The sequence connects the questions, but each lesson uses its own declared experiment. The final evaluation lesson compares fixed controllers; it does not claim to evaluate a trained model carried over from the earlier lessons.

Sources and further study

• Russ Tedrake, Imitation Learning. MIT course notes introduce demonstrations, supervised policy fitting, and observation histories.
• Ross, Gordon, and Bagnell, DAgger (2011). Algorithm 3.1 specifies interactive labeling, aggregation, and expert-policy mixing.
• Ng and Russell, Algorithms for Inverse Reinforcement Learning (2000). The paper examines rewards consistent with observed behavior; its author-hosted summary states the ambiguity directly.
• Ho and Ermon, Generative Adversarial Imitation Learning (2016). The occupancy-matching formulation and Algorithm 1 explain its discriminator and learner-rollout loop.
• Torabi, Warnell, and Stone, Behavioral Cloning from Observation (2018). The authors infer missing actions using an inverse dynamics model learned from the agent's own experience.