explainer
Behavior cloning: learn robot actions from demonstrations
Learn behavior cloning with an interactive robot example and downloadable Python exercise. Split episodes, fit a policy, and compare action error with rollouts.
What you will learn
- Identify the observations and actions in an expert demonstration.
- Predict an action with a nearest-neighbor policy.
- Separate action prediction error from success during a rollout.
- Explain what additional recovery examples change and what they leave unresolved.
- Describe how DAgger collects expert labels on visited states.
- Run an episode-split training and evaluation workflow from saved files.
Before you start
A learner can copy every demonstrated action correctly and still miss its task. In this example, a small robot learns actions near its goal. Start it farther away, and those copied actions move it too slowly to finish within five updates.
You will inspect the examples, calculate prediction error, and follow the resulting motion. Then you will add recovery demonstrations and test a remaining failure. Every number comes from a deterministic teaching model that needs no robot hardware or training service. For a complete file-based workflow, jump to the behavior cloning Python exercise.
Define behavior cloning
Behavior cloning learns a policy through supervised learning on expert demonstrations. A policy maps an observation to an action: u = π(o). The observation could contain joint positions, camera images, or a history of measurements.
An expert supplies pairs of observations and chosen actions. The learner tries to match those actions on similar inputs. Russ Tedrake's MIT notes introduce this as direct policy learning from demonstrations, including regression when the state and action are known.
For a deterministic policy with continuous actions, a common objective is mean squared error:
L(θ) = (1/N) Σᵢ ‖πθ(oᵢ) − uᵢ*‖²
The star marks the expert's action. A neural policy can adjust parameters θ with gradient descent. Our learner stores examples and copies the closest one, so its fitting procedure needs no optimizer.
When demonstrations omit those action labels, behavior cloning from observation adds an inference step. Its inverse model learns from the learner's own labeled interactions before assigning actions to expert state transitions.
Give the learner a small robot model
Imagine a point robot on a line. Its position x is measured in meters, and its goal is x = 0. Each action u commands a displacement in meters:
x[k+1] = x[k] + u[k]
u*[k] = −x[k]
The expert reaches zero in one update: x + (−x) = 0. At x = 0.8 m, it commands −0.8 m. A zero action keeps the robot where it is.
The observation equals the true position, with no measurement noise or hidden state. The model assumes exact displacement execution, no obstacles, and no velocity or acceleration limits. Real feedback control must account for sensing and actuation; here we isolate the learned action rule.
We allow five updates and define success as |x[5]| ≤ 0.025 m. This is a chosen task budget and tolerance. The model specifies positions at update boundaries, with no travel duration or path between them.
Collect and separate expert episodes
Each training episode starts at a chosen position, records one expert action, and ends at zero. The initial dataset contains three episodes:
| Observed start x (m) | Expert action u* (m) | Next position (m) |
|---|---|---|
| −0.10 | +0.10 | 0 |
| 0.00 | 0.00 | 0 |
| +0.10 | −0.10 | 0 |
The zero-start episode records the expert holding the goal. We exclude repeated terminal observations from the other episodes. That keeps the stored examples easy to inspect.
Four separate one-action episodes provide held-out queries: −0.80, −0.08, +0.08, and +0.80 m. Their expert labels also equal −x. Neither training set in this lesson contains those starting positions.
For an evaluation on unseen recordings, assign whole episodes to training or evaluation before extracting frames. Adjacent frames from one recording can share nearly identical observations. The robot-data splitting lesson compares both protocols and shows why a same-episode interpolation score answers a different question.
This tiny held-out set supports a diagnostic comparison. We chose it while designing the lesson, so its score does not estimate general robot performance or serve as an untouched final benchmark.
Copy the closest demonstrated action
Our one-nearest-neighbor policy finds the demonstrated position closest to the current observation. It then copies that example's action:
j = arg minᵢ |x − xᵢ|
π(x) = uⱼ*
At x = 0.08 m, the closest example is x = 0.10 m. The learner commands −0.10 m, while the expert would command −0.08 m. The squared action error is (−0.10 + 0.08)² = 0.0004 m².
At every training position, the learner selects that exact example. Its training error is zero. Between examples, it produces a staircase of constant actions; outside the demonstrated range, it keeps copying the nearest endpoint action.
Equal-distance ties select the lower demonstrated position. The implementation treats distance differences within 10⁻¹² m as ties to make decimal boundary cases repeatable. At x = 0.05 m, the examples at 0 and 0.10 m tie, so the selected action is zero.
Measure error on held-out actions
The two near-goal queries, ±0.08 m, each contribute 0.0004 m². That small error describes inputs close to the training examples. The outer queries expose a different result.
At x = 0.80 m, the learner still copies the action −0.10 m. The expert's action is −0.80 m. The squared error is (−0.10 + 0.80)² = 0.49 m²; the negative outer query gives the same error.
Across all four held-out episodes:
MSE = (0.0004 + 0.0004 + 0.49 + 0.49)/4 = 0.2452 m²
Report the two groups as well as their average. A score drawn only from near-goal states would leave the outer failure untested. The lesson's chart shades the span of training positions, but that span still contains unseen states and gives no accuracy guarantee.
Run the policy on its own observations
A rollout repeatedly measures the current state, predicts an action, and updates the environment. Each action changes the next observation. From the held-out start x[0] = 0.8 m, the narrow learner follows:
| Update k | Learner position (m) | Expert position (m) |
|---|---|---|
| 0 | 0.80 | 0.80 |
| 1 | 0.70 | 0.00 |
| 2 | 0.60 | 0.00 |
| 3 | 0.50 | 0.00 |
| 4 | 0.40 | 0.00 |
| 5 | 0.30 | 0.00 |
The learner ends 0.30 m from the goal and misses the tolerance. It moves toward the goal throughout this rollout. This example demonstrates slow recovery outside training coverage, with failure under a finite update budget.
The DAgger paper by Ross, Gordon, and Bagnell studies the broader sequential-learning issue: actions change the distribution of future observations. Errors can lead to unfamiliar states and further errors. This particular rollout does not diverge, and it does not establish that all behavior-cloned policies fail outside their demonstrations. Use a declared episode-evaluation protocol to compare task outcomes across a wider set of conditions.
The covariate-shift experiment compares expert and learner rollouts from identical starts, then tracks how their visited states separate.
Add demonstrations from farther away
Add four expert episodes starting at −0.90, −0.50, +0.50, and +0.90 m. Their actions still equal −x. We keep the model and held-out queries unchanged.
From x = 0.80 m, the nearest example now starts at 0.90 m. The learner commands −0.90 m, reaching −0.10 m. It then copies the +0.10 m recovery action and reaches zero on update 2.
The outer held-out error drops to (−0.90 + 0.80)² = 0.01 m². The near-goal error stays at 0.0004 m². Their combined mean becomes 0.0052 m², while training error remains zero.
DAgger adds a data-collection loop: execute a policy, obtain expert labels for visited states, aggregate those examples, and train again. Its Algorithm 3.1 and analysis specify the procedure and assumptions behind its guarantees. Our fixed recovery dataset illustrates the value of additional coverage; it does not implement DAgger's iterative collection procedure.
The imitation-learning overview places supervised action fitting alongside interactive labeling, inverse reinforcement learning, and adversarial imitation.
Continue to the DAgger lesson to collect labels on learner-visited states, aggregate them, and compare against additional expert demonstrations at an equal label budget.
Keep the remaining limits visible
Set the start to 0.05 m. Both datasets choose the zero action and stay there. They miss the 0.025 m tolerance even after adding recovery examples.
This failure comes from the nearest-neighbor rule's coarse action choices. More suitable examples or a different policy class could change it. For this exact toy expert, linear regression with slope and intercept can recover u = −x from the three noiseless training pairs and reach zero from every allowed start.
That result depends on the expert actually being linear and the model executing actions exactly. A camera policy with ambiguous observations needs different assumptions. Real demonstrations also carry choices about sensor history, units, action timing, and which situations the operator visits.
The two recovery outcomes answer specific questions: the 0.80 m rollout improves, and the 0.05 m rollout still fails. Neither gives a safety guarantee, a statistical success rate, or evidence from a physical robot.
Inspect the learned policy and rollout
Start with the default near-goal dataset and a rollout from 0.80 m. Check its zero training loss alongside the 0.30 m final distance. Move the replay to update 0 to inspect the first copied action.
Switch to Recovery demonstrations added. The policy chart gains new steps, and the rollout reaches zero after overshooting to −0.10 m. Then try a start of 0.05 m to expose the remaining zero-action region.
The readouts show the state after the selected number of updates and preview the next action. At update 5, that preview lies beyond the experiment's budget.
Reproduce the example in Python
This standard-library script stores demonstrations, predicts actions, and executes the same five updates. It also keeps the held-out queries separate from training.
sets = {
"narrow": [-0.1, 0.0, 0.1],
"recovery": [-0.9, -0.5, -0.1, 0.0, 0.1, 0.5, 0.9],
}
held_out = [-0.8, -0.08, 0.08, 0.8]
def predict(x, demos):
distances = [abs(x - s) for s in demos]
best = min(distances)
nearest = min(
s for s, d in zip(demos, distances) if d <= best + 1e-12
)
return -nearest
def mse(starts, demos):
return sum((predict(x, demos) + x) ** 2 for x in starts) / len(starts)
def rollout(start, demos):
states = [start]
for _ in range(5):
states.append(states[-1] + predict(states[-1], demos))
return states
for name, demos in sets.items():
states = rollout(0.8, demos)
print(f"{name}: train MSE={mse(demos, demos):.6f}, "
f"held-out MSE={mse(held_out, demos):.6f}")
print("positions: " + ", ".join(
f"{(0 if abs(x) < 1e-12 else x):.2f}" for x in states
))
print(f"final distance={abs(states[-1]):.6f}, "
f"goal={abs(states[-1]) <= 0.025 + 1e-12}")
print(f'near-goal MSE={mse([-0.08, 0.08], sets["narrow"]):.6f}')
print(f'recovery from 0.05: {rollout(0.05, sets["recovery"])[-1]:.6f}')
Expected output:
narrow: train MSE=0.000000, held-out MSE=0.245200
positions: 0.80, 0.70, 0.60, 0.50, 0.40, 0.30
final distance=0.300000, goal=False
recovery: train MSE=0.000000, held-out MSE=0.005200
positions: 0.80, -0.10, 0.00, 0.00, 0.00, 0.00
final distance=0.000000, goal=True
near-goal MSE=0.000400
recovery from 0.05: 0.050000
Behavior cloning Python example: split, train, and evaluate
The download connects this lesson to episode splitting and policy evaluation. It contains a synthetic demonstration dataset, saved split manifests, a fitted policy, and the code to reproduce them. Python 3.12 or later is enough; no additional packages are required.
This is a second robot model. The earlier widget commands displacement in meters. Here the observation is position x in meters and velocity v in meters per second. The action u commands acceleration in meters per second squared, clipped to [−1, 1]. Every update lasts Δt = 0.2 seconds:
a = g u + d
x[k+1] = x[k] + v[k] Δt + ½ a Δt²
v[k+1] = v[k] + a Δt
The expert commands clip(−2x − 1.8v, −1, 1). Demonstrations use actuator gain g = 1 and disturbance d = 0, with exact observations. Each episode starts from x drawn uniformly in [−0.8, 0.8] m and v in [−0.15, 0.15] m/s. The fixed seed produces 80 episodes of 30 actions, or 2,400 labeled rows. Collection continues after reaching the goal, so these records include many easy near-goal actions. Rollout evaluation instead stops at its first outcome.
Download and run the exercise
Download the complete behavior-cloning exercise ZIP. Unzip it and open a terminal in the extracted directory. The README explains the files, environment, and checks; bc.py contains the implementation, and expected-output.txt records the default check output.
To inspect the supplied data before running anything, open the episode CSV, dataset description, and training manifest. These are generated teaching demonstrations, not recordings from a physical robot. The script validates this declared expert and model; it is not a general-purpose demonstration importer.
python3 bc.py generate --out run
python3 bc.py split --out run
python3 bc.py train --out run
python3 bc.py evaluate --out run
python3 bc.py check > check-output.txt
diff -u expected-output.txt check-output.txt
Use a fresh run directory. The first command writes the demonstration CSV and reset conditions. The next three commands consume those saved files: split complete episodes, fit and select a policy, then evaluate that frozen policy. check separately regenerates the default exercise in a temporary directory and tests its invariants, and compares the supplied reference/ artifacts when present. The final comparison should print nothing and exit successfully. To validate and reevaluate your saved run, use evaluate --out run again; it does not refit the policy.
Fit three candidates using training episodes only
The manifests assign 48 episodes to training, 16 to validation, and 16 to final testing. All 30 rows from an episode stay together. Training observations determine the position/velocity means and standard deviations; the same transformation is then applied to every split.
The candidate policies are small enough to inspect:
- Affine: fit an intercept and two slopes to the standardized observations by least squares, with a fixed ridge penalty of 0.000001 on the slopes. Clip the predicted action to the allowed range.
- One neighbor: copy the training action nearest to the standardized observation.
- Seven neighbors: average the actions of the seven nearest training observations. Equal distances follow training CSV order.
The affine fit minimizes squared error before output clipping. Validation evaluates the clipped predictions for all candidates, using mean squared action error across its 480 records. The lowest score wins; an exact tie follows the listed candidate order. Neither final-test labels nor rollout success select the model.
| Candidate | Validation action MSE, (m/s²)² |
|---|---|
| Affine | 0.00044645 |
| One neighbor | 0.00094054 |
| Seven neighbors | 0.00087992 |
The default run freezes the affine policy in run/model.json. Its selection record saves the scores and hashes identifying the model, dataset, and split membership. Changing final labels in the fitting check leaves the selected model unchanged. This tests the code's data flow; the supplied teaching results were inspected while authoring the exercise, so they are not an untouched benchmark.
Read the action score alongside the rollout
On the 480 final demonstration records, action MSE is 0.00044231 (m/s²)². Restricting the calculation to the first five actions of each episode raises it to 0.00184831, because the full score gives substantial weight to the later near-goal records. Both calculations describe actions at expert-visited states.
The learned policy succeeds on 16 of 16 nominal rollouts and 14 of 16 rollouts with changed dynamics. Each action is chosen from the learner's current state, and that action determines the next state. The expert reference succeeds on 16 and 15, respectively, under the same paired conditions. These are small synthetic diagnostic batches, not evidence of hardware performance or a general success probability.
Continue with the actual split and preprocessing boundary, then inspect the evaluation conditions, stopping rules, and per-episode results. Those sections use this same download.
Try it yourself
Exercise 1. Use the recovery dataset and start at −0.80 m. Find the first two actions, the two resulting positions, and the first action's squared prediction error.
Show the recovery calculation
The nearest demonstration starts at −0.90 m, so the first action is +0.90 m. The robot reaches +0.10 m, then chooses −0.10 m and reaches zero. The expert's first action would be +0.80 m, giving squared error (0.90 − 0.80)² = 0.01 m².
Exercise 2. Both policies remain at +0.05 m under the stated tie rule. Add one expert episode at +0.05 m to the narrow training set. What changes for that rollout, and does its success establish performance at every other start?
Show the added-example result
The new label is −0.05 m. The learner selects that exact example and reaches zero in one update. This checks one start; other queries still follow their nearest examples, and they require their own evaluation. Keep any final evaluation episodes separate when choosing additional demonstrations.
A policy can also encounter a different physical response after training. The sim-to-real transfer lesson changes a simulated actuator gain and compares controllers fitted using one model or several. It makes the model-mismatch question visible before considering what a hardware test would need to establish.
A diffusion policy can represent several plausible demonstrated actions and generate an action sequence through denoising. Vision-language-action models add visual and language inputs; their outputs still need defined units, coordinate frames, and timing before a robot can use them.
The person who supplies demonstrations can play a different role during deployment. Autonomy and teleoperation separates training data collection from the decisions and interventions needed during a robot run.
Sources and further study
- robomimic: dataset contents: trajectory organization and environment metadata in a robot-learning dataset. This lesson uses its own small CSV format.
- Russ Tedrake, Underactuated Robotics: Imitation Learning: supervised policy learning, observation/action choices, and the distinction between state and observation-history inputs.
- Ross, Gordon, and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2011): learner-induced state distributions and DAgger. The linked paper gives the algorithm and the assumptions required for its analysis.
To change the policy class in this example, work through linear regression. Fit the three demonstration pairs, then compare its actions with the nearest-neighbor staircase at the same held-out starts.