# Behavior cloning: a complete episode-to-rollout exercise

This original synthetic exercise connects episode splitting, supervised policy fitting, validation-only model selection, final action error, and closed-loop evaluation. It uses Python 3.12+ and the standard library. No downloads, training service, robot hardware, or third-party Python packages are required.

The supplied default run and checks were verified on Python 3.14.3 on macOS ARM64 and Python 3.12.3 on Ubuntu 24.04.5 ARM64. Both environments reproduced every supplied reference artifact byte for byte.

The site lessons keep their existing interactive examples. This download introduces a separate experiment: it uses the acceleration plant and stopping-rule structure from the policy-evaluation lesson, with a new dataset, new episode conditions, and a learned controller. It does not reproduce the displacement model in the behavior-cloning widget or the episode-offset model in the data-leakage widget.

## Run the five stages

Extract the ZIP into a directory, open a terminal there, and run:

```sh
python3 bc.py generate --out run
python3 bc.py split --out run
python3 bc.py train --out run
python3 bc.py evaluate --out run
python3 bc.py check
```

`generate` defaults to seed 7 and requires an empty output directory. Use a fresh directory for a new dataset, for example `python3 bc.py generate --out run-23 --seed 23`. Later stages read the saved seed; they cannot silently change it. Repeat the split, train, and evaluate commands with that new directory.

The ZIP includes a complete seed-7 run in `reference/`, including the CSV dataset, all three split manifests, selected model, and evaluation traces. Inspect these files directly; generate your own `run/` directory to leave the supplied reference unchanged.

`check` runs numerical and data-boundary checks against a freshly generated default experiment in a temporary directory. When the supplied `reference/` directory is present beside the script, it also compares all 13 generated artifact files byte for byte with that reference. It does **not** inspect an arbitrary saved run and does not accept `--out`. Compare its complete printed output with the supplied reference:

```sh
python3 bc.py check > check-output.txt
diff -u expected-output.txt check-output.txt
```

No output from `diff` means the rounded printed results match. The check itself verifies a hand-calculated dynamics step, split overlap rejection, training-only normalization, invariance to changed final labels, stopping priorities, metric denominators, file round trips, and dataset hash rejection. Reference-file comparison is stricter than the rounded display. Byte equality is verified for the two CPython environments listed above; other interpreter/build combinations remain unverified and must pass the check before being treated as the same reference run. Python changed [floating-point summation in 3.12](https://docs.python.org/3.12/library/functions.html#sum), so older versions are excluded from this exercise’s numerical reference contract. A changed artifact produces a named error; do not silently accept it as the same reference experiment. If you downloaded only `bc.py`, the reference-file comparison is skipped. These checks are not a proof of generalization or a test of every possible code change.

To check and rerun an existing saved experiment, use `python3 bc.py evaluate --out run`. It verifies the dataset, split, and frozen-model hashes, then regenerates evaluation artifacts without fitting parameters or selecting a model. Re-running `train` is a separate, explicit model-selection step. Once evaluation results influence a modeling choice, that batch is part of development; generate and reserve a new final evaluation for a subsequent claim.

## What each stage writes

| Stage | Files | Purpose |
| --- | --- | --- |
| generate | `episodes.csv`, `conditions.json`, `dataset.json` | Demonstration rows, reset conditions, seed, and input file hashes |
| split | `splits/train.json`, `splits/validation.json`, `splits/final-test.json` | Whole-episode IDs, role, and dataset hash |
| train | `model.json`, `selection.json` | Fitted policy and preprocessing, validation scores, selection rule, and frozen artifact hashes |
| evaluate | `evaluation.json`, `action-errors.csv` | Final action errors and aggregate episode metrics |
| evaluate | `rollout-conditions.json`, `rollout-results.csv`, `rollouts.csv` | Every paired reset and dynamics condition, episode outcome, and executed action/state transition |

`model.json` contains the selected policy's training-only means and standard deviations, plus either its affine coefficients or its nearest-neighbor training examples. The model also records training and validation episode IDs and record digests. `selection.json` records the model file, dataset, and split digests. These hashes detect accidental changes; they are not security controls or a mechanism that makes a test set inaccessible. Training loads and validates the full CSV for integrity, but final-record values do not enter fitting, preprocessing, or model selection.

The CSV has columns `episode_id,step,x,v,action,x_next,v_next`. A row records the state **before** action `step` and the state after executing it. Steps run from 0 through 29. The state after action 29 has no subsequent action label. Each nonfinal next state equals the following row's state in the same episode. The loader rejects missing steps, nonfinite values, broken transitions, nonexpert labels, and unexpected episode IDs in addition to checking input file hashes.

There are **80 episodes and 2,400 action rows**. The splitter orders episode IDs by a seed-keyed SHA-256 digest, then assigns **48 training, 16 validation, and 16 final-test episodes**. It rejects overlapping, missing, duplicate, or unknown IDs. The resulting row counts are 1,440, 480, and 480. Individual frames are never split across roles.

## The declared model and demonstration collection

State is position `x` in meters and velocity `v` in m/s. Observation is the exact full state. Command `u` is an acceleration in m/s², clipped to [-1, 1]. Holding the command for `dt = 0.2 s` gives:

```text
a = gain * u + disturbance
x_next = x + dt * v + 0.5 * dt² * a
v_next = v + dt * a
expert(x, v) = clip(-2*x - 1.8*v, -1, 1)
```

The feedback coefficients have units 1/s² and 1/s, respectively. Demonstrations use gain 1 and disturbance 0. Each episode starts with independent idealized uniform draws `x0 ∈ [-0.8, 0.8] m` and `v0 ∈ [-0.15, 0.15] m/s`. A SHA-256 hash of seed, namespace, and episode ID seeds Python's pseudorandom generator. Mixing that tuple avoids the near-identical initial states that some simple generators produce for consecutive raw seeds. It does not prove statistical independence.

Each demonstration collects **30 actions even after the state first meets the goal condition**. Those later near-goal examples remain in the offline data. This fixed-length collection protocol differs from the rollout evaluator, which stops at its first task outcome. The final offline MSE therefore gives equal weight to all 480 final action rows, including easy holding behavior. Since all episodes have the same number of rows, this also equals the mean of the 16 per-episode MSEs. The report separately gives error over the first five actions of each episode to reveal the effect of that weighting.

## Fit on training episodes; select on validation episodes

All three declared candidates share the same training/validation boundary and training-only feature standardization. Standard deviations use population variance, dividing by the number of training rows. Neither validation nor final inputs enter that calculation.

- **affine:** least-squares action regression on an intercept plus standardized position and velocity. Its objective is mean squared residual plus `1e-6` times the sum of squared standardized slopes. The intercept is unpenalized. The final prediction is clipped to the action range. The fit minimizes the un-clipped affine regression objective; selection scores the clipped predictions.
- **knn-1:** copies the action of the closest training observation in standardized Euclidean distance.
- **knn-7:** averages the actions of the seven closest training observations in the same metric. Equal distances use training CSV order.

A clipped affine rule can represent this expert exactly. The chosen fitting objective, however, minimizes unclipped regression error against saturated expert labels, so it need not recover the expert gains. The nearest-neighbor candidates do not consult future expert observations during a rollout. They predict from the learned policy's current position and velocity.

Selection uses the lowest validation action MSE. Exact floating-point ties use the declaration order affine, knn-1, knn-7. The chosen policy is then serialized and frozen. There is no second fit on training plus validation, and no rollout-driven tuning or final-label-driven selection. All action MSEs have units `(m/s²)²`.

For seed 7, the validation scores are:

| Candidate | Validation action MSE |
| --- | --- |
| affine | 0.00044645 |
| knn-1 | 0.00094054 |
| knn-7 | 0.00087992 |

The affine policy wins this declared selection rule. Its nominal final action MSE is **0.00044231**, while the first-five-actions MSE is **0.00184831**. These are different averages of the final expert-recorded observations. The smaller full-trajectory score alone does not establish successful feedback control.

## Evaluate two closed-loop conditions

The selected policy and the hand-written expert reference each run from the **same 16 final episode resets**. The expert is a reference controller, not an optimality claim. Both policies receive the same gain and disturbance within a paired condition. Their future observations differ because they execute different actions.

1. **Nominal:** gain 1, disturbance 0, matching the demonstration dynamics.
2. **Shifted:** gain drawn uniformly from [0.6, 1.4] and a constant disturbance drawn uniformly from [-0.25, 0.25] m/s², independently per episode. A separate keyed generator namespace fixes these draws. Gain and disturbance stay constant during the episode.

The offline final labels were collected under nominal dynamics. Their MSE is **not** an offline accuracy score for the shifted environment. Shifted outcomes answer the separate task-completion question recorded by the rollout experiment. The paired scenarios reuse resets, so do not treat all 32 resets as independent observations from one distribution.

Evaluation checks the **post-action** state at every 0.2-second sample:

- A sampled position `|x| ≥ 1.25 m` records **boundary failure**, with first priority.
- Three consecutive post-action samples with `|x| ≤ 0.10 m` and `|v| ≤ 0.15 m/s` record **success**. Leaving either band resets the count. The initial reset does not count.
- If neither occurs by action 30, the six-second external budget records **timeout**. Success on the last sample takes priority over timeout.

Success and boundary failure set `terminated=true`; timeout sets `truncated=true`. Three qualifying samples span 0.4 seconds from first to third. These sampled checks do not establish continuous settling or safety between observations.

The default results are:

| Condition/controller | Success | Boundary | Timeout | Mean time among successes |
| --- | --- | --- | --- | --- |
| Nominal learned | 16/16 | 0 | 0 | 2.225000 s |
| Nominal expert | 16/16 | 0 | 0 | 2.125000 s |
| Shifted learned | 14/16 | 0 | 2 | 2.357143 s |
| Shifted expert | 15/16 | 0 | 1 | 2.440000 s |

In the shifted batch, both controllers succeed on 14 episodes, only the expert succeeds on one, and neither succeeds on one. The learner's smaller conditional time in that batch does not make it better: it omits its two timeouts from that mean. Every attempted episode remains in the success-rate denominator. A run with zero successes would report `null` for the conditional mean, displayed as “undefined.” No observed boundary failures is not a safety guarantee.

In shifted `episode-016`, the learned policy finishes inside both success bands but has not accumulated three consecutive qualifying samples before the budget expires. It therefore times out. In shifted `episode-014`, both controllers time out outside the position band. Those are different reasons for missing the declared task condition.

Inspect `rollout-results.csv` for episode IDs and endings. Join an ID and scenario to `rollout-conditions.json` and `rollouts.csv` to reproduce each transition. The learned policy uses its own evolving state, never the next recorded expert state.

## Scope and further experiments

These are authored, deterministic teaching results from a small synthetic batch. We examined the default results while developing the exercise, so “final-test” describes its software data-flow role, **not an author-blind, untouched benchmark**. The pipeline keeps final data out of fitting and selection within a run. It cannot erase earlier human choices informed by those results.

The model has exact observations, no delay, no contact, and a simple actuator. Results say nothing directly about hardware, camera inputs, objects, real demonstration quality, or population-level success. Sixteen episodes provide a readable diagnostic, not a precise generalization estimate.

Try these changes in a new directory and record what you changed before inspecting final results:

1. Change the declared candidate set or training data-collection range in `bc.py`; keep selection on validation episodes. Report both the validation rule and resulting frozen policy.
2. Inspect the first-five-actions and full-trajectory errors together. Try a different **predeclared** weighting rule for validation selection and use a new final batch for a later claim.
3. Add a new rollout condition without changing the frozen policy, then label it as an additional diagnostic. If you adapt the policy after seeing it, it becomes development evidence.

The generator is the source of the demonstration CSV. To use a different expert or model, edit its generation and validation contracts together and generate a fresh run. Do not change artifact hashes merely to disguise a modified dataset as the original.

Related lessons: [behavior cloning](https://www.dustinturner.ai/writing/behavior-cloning), [robot data leakage](https://www.dustinturner.ai/writing/robot-data-leakage), and [robot policy evaluation](https://www.dustinturner.ai/writing/robot-policy-evaluation).