Robot data leakage: split demonstrations by episode

A robot policy can score well on held-out frames while failing on held-out episodes. Frames from one demonstration share a scene, a task setup, and nearby moments. A split that mixes those frames answers a different question from a split that holds out an entire run. After the example, inspect a downloadable three-way episode split used to train and evaluate a policy.

This lesson builds both splits, fits the same learner, and checks every prediction. The data are synthetic and deliberately expose an episode-specific pattern. They provide no estimate of any real robot's performance.

Start with the deployment question

Write the question before choosing the split: How well will this fitted predictor copy actions in an episode it has never seen? For that question, hold out the complete evaluation episode, including its action labels.

A different question could ask how well a model predicts unlabelled frames in an already recorded episode. An interleaved-frame evaluation can serve that offline task when the available context matches the intended use. A valid score for that task does not become label leakage merely because nearby frames correlate.

The failure occurs when a report presents same-episode prediction as evidence about unseen episodes. The training set then contains context and labels from episodes that the stated protocol promised to withhold. Also distinguish this protocol mismatch from directly fitting on test labels or choosing a model after repeatedly checking a final test set.

Preserve episodes when you store frames

The RLDS dataset format stores episodes containing steps and supports episode-level metadata. Keeping those boundaries lets you group observations before making training windows or action chunks.

Our fixed dataset, synthetic-episodes-v1, contains six episodes, A through F. Each supplies six labeled frames, numbered 0 through 5. All 36 records contain valid observation/action pairs; the example includes no terminal-only state.

For episode index e = 0, …, 5 and frame f = 0, …, 5, construct:

Here x is a supplied, dimensionless scene descriptor that changes slowly during a run. Think of a stable appearance measurement that the sensor also provides at prediction time. We assign its episode baselines 0 through 5 to keep the example reproducible; the learner receives the measured scalar, not the episode index or a hidden label.

The normalized action baseline b alternates between −0.6 and +0.6:

Episode	Descriptor baseline	Action baseline
A	0	−0.6
B	1	+0.6
C	2	−0.6
D	3	+0.6
E	4	−0.6
F	5	+0.6

This alternating pattern is an authored test case. It makes nearby frames informative about their own episode while making neighboring episodes poor substitutes. The scalar descriptor is a legitimate input; its availability does not establish that distance in this feature predicts action similarity across scenes.

Compare two equal-size split rules

Both protocols train on 24 records and score 12 other records. No record ID appears on both sides.

• Interleaved frames: hold out frames 2 and 5 from every episode. Train on frames 0, 1, 3, and 4. All six episodes occur on both sides.
• Whole episodes: hold out B and E, including all their frames. Train on A, C, D, and F. No episode occurs on both sides.

The frame protocol also places later frames in training while scoring earlier frames. Frame 2 lies between training frames; frame 5 sits just after the last training frame. Its intended task is offline prediction within known episodes; it does not represent forecasting from past information alone.

Scikit-learn's grouped-validation guide explains how group-aware splits keep each group from appearing in both training and testing within a fold. Our widget uses three explicit episode pairs: A/D, B/E, and C/F. Across those folds, each episode contributes held-out records once.

Choose the grouping level that matches deployment. Episode separation alone does not establish generalization to a new room, object instance, operator, or collection day if those factors still span both sets. A temporal deployment question may also require a chronological boundary. The robot-learning generalization experiment makes the held-out conditions visible and separates unseen combinations from new values.

Fit the predictor on training records only

The learner uses one-nearest-neighbor regression. Fitting stores the training observations and action labels. For a query x, it finds the training row with the smallest absolute observation difference and copies that row's action:

Distances within 10⁻¹² count as a numerical tie; the lower record ID wins. This rule makes the example deterministic. Episode metadata never enters the distance calculation.

The predictor receives no held-out actions during fitting or prediction. It also performs no learned scaling, feature selection, or preprocessing. The toy's feature-to-action relation and the model's distance rule remain fixed across both protocols.

At a training observation, the model finds that exact record and returns its label. Zero training error therefore says little about its performance on a new episode. The behavior-cloning lesson develops the same distinction between fitting demonstrations and evaluating other observations.

Calculate the error for one held-out record

In the frame split, record A2 has x = 0.02 and action −0.56. Training records A1 and A3 both lie 0.01 away. The tie rule chooses A1, whose action is −0.58.

Every held-out frame has this same signed error. The framewise mean squared error, or MSE, is 0.000400. Its root, RMSE, is 0.020000 normalized action units.

Now hold out B and E. Record B0 has x = 1.00 and label +0.60. Its closest training observation is A5 at x = 0.05, with action −0.50:

Across all 12 held-out records, squared errors sum to 17.36. Thus MSE = 17.36 / 12 = 1.446667, and RMSE = 1.202775. The two protocols score different records; their gap is a result for this constructed dataset, not a universal correction factor for leakage.

Inspect membership and predictions

Start with Interleaved frames. The membership grid shows training and held-out records in every episode. Read the A2 prediction, then move the record control to inspect other held-out frames.

Switch to Whole episodes. Shared episodes drops from six to zero, and the default probe becomes B0. Change the episode pair to check the other two folds while preserving the same training and scoring counts.

Robot data leakage

Which episodes did the learner already see?

Compare two fixed splits of the same 36 records. Each trains a one-nearest-neighbor model on 24 records and scores 12 held-out records.

Evaluation splitInterleaved framesWhole episodes

Held-out episode pairA and DB and EC and F

Held-out record

1 of 12: A2

Reset split experiment

The episode-pair control changes the whole-episode comparison, even when the grid shows interleaved frames. No held-out label enters fitting. The declared deployment question is prediction in a previously unseen episode.

Keep the episode boundary visible

Compare each prediction with its label

Training records

24

Held-out records

12

Training episodes

6

Held-out episodes

6

Shared episodes

6

Interleaved-frame MSE

0.000400

Whole-episode MSE

1.446667

Selected split RMSE

0.020000

Selected record

A2

Observed feature

0.020000

Nearest training record

A1

Recorded action

-0.560000

Predicted action

-0.580000

Prediction error

-0.020000

All six episodes appear in training and scoring. This split checks prediction within known episodes; reporting it as unseen-episode performance misstates the evaluation. MSE uses squared normalized action units; RMSE and signed prediction error use normalized action units.

Inspect the selected split manifest

Dataset: synthetic-episodes-v1. Learner: one nearest neighbor on the observed scalar; distance ties within 1e-12 choose the lower record ID. No fitted preprocessing.

Training IDs: A0, A1, A3, A4, B0, B1, B3, B4, C0, C1, C3, C4, D0, D1, D3, D4, E0, E1, E3, E4, F0, F1, F3, F4

Held-out IDs: A2, A5, B2, B5, C2, C5, D2, D5, E2, E5, F2, F5

The action chart uses the same fixed vertical scale in every view. Its horizontal positions enumerate held-out records; they do not represent a continuous rollout. The widget recomputes both split scores whenever the episode pair changes.

Say what the score actually measures

The default interleaved score measures action prediction on unseen frames from known episodes. The B/E score measures action prediction on two specific unseen episodes. Neither score measures task success, collision rate, or recovery after the learner makes a mistake. The policy-evaluation lesson runs complete episodes and counts successes, failures, and timeouts under a declared protocol.

Our three whole-episode folds give MSEs 1.385167, 1.446667, and 1.385167. Six episodes provide a small, deliberately patterned demonstration. Thousands of correlated frames would not turn them into thousands of independent runs.

Equal episode lengths make the frame-averaged and episode-averaged MSE identical here. With unequal lengths, report how you aggregate: pooling every frame gives longer episodes more influence. Keep per-episode errors available so that one long easy run cannot conceal several failed runs.

A clean episode split also leaves behavior-cloning covariate shift unresolved. The learner's actions can lead it to observations absent from every demonstration. Evaluating that effect requires a rollout with the learned policy.

Save a reproducible split manifest

The widget's disclosure lists every training and held-out record ID. A real experiment also needs the dataset version, grouping rule, feature definition, model settings, and metric convention. Save actual IDs alongside the generation rule so later dataset reordering cannot silently change the split.

For the default whole-episode fold, the core manifest reads:

{
  "dataset": "synthetic-episodes-v1",
  "group_key": "episode",
  "train_episodes": ["A", "C", "D", "F"],
  "held_out_episodes": ["B", "E"],
  "frames_per_episode": [0, 1, 2, 3, 4, 5],
  "feature": "observed scalar scene descriptor",
  "target": "normalized action",
  "learner": "one-nearest-neighbor",
  "distance": "absolute feature difference",
  "tie_rule": "within 1e-12, lower record ID",
  "learned_preprocessing": "none",
  "metric": "mean squared action error over held-out records"
}

Assign whole episodes to splits before producing overlapping observation windows, augmented copies, or action chunks. Keep every derived sample with its source episode. If deployment requires a stronger grouping boundary, record that boundary as well.

Keep preprocessing and tuning inside the boundary

Scikit-learn's common-pitfalls guide describes a separate leakage path: fitting transformations with information from held-out data. For example, fit a scaler's mean and variance on the training subset, then apply that fixed transform to both subsets. A pipeline can repeat the fitting boundary within each cross-validation fold.

That preprocessing failure is not simulated in this widget. Here, both learners fit only their assigned training records. The lesson's numerical difference comes from the episode membership of those records.

Use development folds to choose features, model settings, or which demonstrations to add. Preserve a final evaluation set that does not guide those choices. These transparent toy folds are inspectable teaching examples, not a sealed benchmark.

When DAgger collects expert labels at learner-visited states, mark those episodes as training data. Do not later report those same episodes as an independent final evaluation. A split manifest makes that history reviewable.

Reproduce the splits and scores in Python

This standard-library script builds all records, fits through stored training rows, and calculates the scores. Integer feature and action coordinates keep the nearest-neighbor calculation exact before converting to normalized units.

from math import sqrt

rows = [
    {"id": f"{chr(65 + e)}{f}", "episode": e, "frame": f,
     "x100": 100 * e + f, "a100": (-60 if e % 2 == 0 else 60) + 2 * f}
    for e in range(6) for f in range(6)
]


def evaluate(mode, pair):
    def held_out(row):
        if mode == "frames":
            return row["frame"] in (2, 5)
        return row["episode"] in pair

    train = [dict(row) for row in rows if not held_out(row)]
    test = [row for row in rows if held_out(row)]
    predictions = []
    for row in test:
        nearest = min(train, key=lambda r: (
            abs(r["x100"] - row["x100"]), r["id"]
        ))
        error = (nearest["a100"] - row["a100"]) / 100
        predictions.append((row["id"], nearest["id"], error))
    mse = sum(p[2] ** 2 for p in predictions) / len(test)
    shared = {r["episode"] for r in train} & {r["episode"] for r in test}
    return mse, sqrt(mse), len(shared), predictions[0]


for name, mode, pair in [
    ("frames", "frames", (1, 4)),
    ("episodes B/E", "episodes", (1, 4)),
    ("episodes A/D", "episodes", (0, 3)),
    ("episodes C/F", "episodes", (2, 5)),
]:
    mse, rmse, overlap, first = evaluate(mode, pair)
    print(f"{name}: MSE={mse:.6f}, RMSE={rmse:.6f}, shared={overlap}")
    if name in ("frames", "episodes B/E"):
        print(f"  {first[0]} copies {first[1]}: error={first[2]:.6f}")

Expected output:

frames: MSE=0.000400, RMSE=0.020000, shared=6
  A2 copies A1: error=-0.020000
episodes B/E: MSE=1.446667, RMSE=1.202775, shared=0
  B0 copies A5: error=-1.100000
episodes A/D: MSE=1.385167, RMSE=1.176931, shared=0
episodes C/F: MSE=1.385167, RMSE=1.176931, shared=0

Inspect the downloadable episode split

The shared behavior-cloning exercise puts the fitting boundary into a complete workflow. Its acceleration-controlled point robot is a separate task from the scene-descriptor example above. It supplies 80 expert episodes, each containing 30 position/velocity observations and acceleration labels. Download and run the package from the behavior-cloning lesson, then inspect these files:

Split	Episodes	Action rows
Train	48	1,440
Validation	16	480
Final test	16	480

Training fits normalization and each candidate policy. Validation selects among the three declared candidates. Final testing reports the frozen policy's action error and rollouts.

Each CSV row contains episode_id, step, x, v, action, x_next, and v_next. A row records the state before an action and the resulting state after it. Consecutive rows within an episode must agree on that transition. The final row still has a valid action; it is not a terminal observation with a fabricated label. Collection uses a fixed horizon and continues after reaching the goal.

Follow the boundary through the code

split deterministically orders episode IDs using the dataset seed, then assigns whole episodes to the three manifests. It rejects overlapping membership, missing IDs, and incomplete partitions. It does not shuffle individual frames across those boundaries. Episode separation asks about other starts in this same synthetic task; it does not establish transfer to other rooms or robot types.

train first extracts the 1,440 training records. For each feature, it computes the training mean μ and population standard deviation σ, then transforms observations using z = (value − μ) / σ. Those fixed values transform validation and final observations too. Refitting σ on all 2,400 rows would let the final observations influence the model, even without reading their action labels.

The candidate fits receive only training records. Validation labels choose between the affine, one-neighbor, and seven-neighbor policies. Final rows have no role in that choice. The automated fitting check changes final action labels and verifies that the selected policy remains identical; file-integrity checks separately reject an edited dataset during the normal command workflow.

Keep the result attached to its inputs

The dataset record identifies the CSV and reset conditions with SHA-256 hashes. Each split manifest identifies its CSV, and the selection record identifies the frozen model and split membership. evaluate rejects a mismatched dataset, split, or model. A hash detects a changed artifact; it cannot establish that the data or experiment design is good.

For a planned training change, use a separate output directory and retain the original files for comparison. If you choose the next model because of its final score, that score has become development feedback. A new final evaluation then needs data or conditions that did not inform the choice. These author-inspected examples demonstrate the code boundary without claiming a blind benchmark.

Next, follow the frozen model into paired closed-loop evaluation. That step explains why a valid split and a low action error still leave a separate task-success question.

Check the evaluation protocol

Exercise 1. Under the B/E whole-episode split, find the nearest training row for E5. Calculate its prediction, signed error, and squared error. Does the frame split give the same prediction for E5?

Show the E5 calculation

E5 has x = 4.05 and label −0.50. In the whole-episode split, F0 at x = 5.00 is closer than D5 at x = 3.05. F0 supplies +0.60, giving error +1.10 and squared error 1.21. In the frame split, E4 at x = 4.04 supplies −0.52, giving error −0.02 and squared error 0.0004. Same-episode training context changes the prediction.

Exercise 2. You want performance on new episodes from a new collection room. You split episodes without overlap, but every room appears in training and evaluation. You also choose the final model by checking evaluation MSE after each change. Identify two remaining problems and a protocol that addresses them.

Show the protocol repair

The split tests new episodes from represented rooms, so it does not isolate the new-room question. Repeated model selection also lets evaluation results influence the chosen model. Group the final holdout by room, keep each room's episodes and derived windows together, and use separate development groups for model selection. Fit preprocessing on each training subset. A final score then addresses those held-out rooms under the stated collection conditions; broader claims still need further evidence.

Sources and next steps

• Scikit-learn: cross-validation for grouped data, including the distinction between sample-wise splits and held-out groups.
• Scikit-learn: common pitfalls and data leakage, covering training-only fitting and pipelines.
• Google Research: RLDS dataset format, describing episode, step, and metadata structure.

Next, inspect behavior-cloning covariate shift. Keep the offline split fixed, then ask what happens when the learner's own actions change the observations it must handle.