Robot policy evaluation: success, failure, and episode counts

For an exercise that connects training and evaluation, jump to the downloaded learned-policy results. The fixed-controller example below introduces the evaluation rules first.

Ask what a complete episode proves

Robot policy evaluation measures what happens when a policy controls a robot through a task. The result depends on the success rule, starting conditions, stopping rules, and number of attempted episodes.

An action-prediction score answers a narrower question. A policy with low behavior-cloning loss still needs rollout tests, because its actions change the states it visits. The covariate-shift experiment shows how those two measurements can disagree. The replay versus interactive simulation experiment isolates the evaluation loop: the same feedback policy reads either fixed recorded states or states caused by its own commands.

The robomimic evaluation configuration exposes rollout count, horizon, and termination on success. Those choices belong in the evaluation report. Here, two fixed controllers make the choices inspectable without training a model.

Define success before running the policy

Our simulated robot must reach rest near x = 0. We check its true position and velocity every 0.2 seconds, after applying an action.

• Success: three consecutive post-action samples satisfy |x| ≤ 0.10 m and |v| ≤ 0.15 m/s.
• Boundary failure: a sampled position satisfies |x| ≥ 1.25 m.
• Timeout: the external evaluation budget expires before either task outcome.

A sample outside either success band resets the consecutive-sample count. The initial reset does not count. Three qualifying samples span 0.4 seconds from the first to the third; they do not prove continuous settling between samples.

The simulator stops at the first outcome. It gives boundary failure priority, then success, then timeout. A success on the final allowed sample counts as success.

These rules define a small sampled benchmark. A physical robot would need a task-specific success detector and safety monitoring that covers motion between observations. This lab makes no continuous safety guarantee.

Specify the simulated robot and controllers

The state contains position x in meters and velocity v in m/s. An action u commands acceleration in m/s². The actuator applies acceleration a = gu + d, where g is a dimensionless gain and d is a constant episode disturbance in m/s².

Holding an action for Δt = 0.2 s gives exact endpoints:

Both controllers observe x measured = x + ε and the exact velocity. They use this saturated feedback law:

The position coefficient has units 1/s². The velocity coefficient has units 1/s: k_d = 0.2 for the baseline and 1.8 for the damped controller. These are fixed feedback controllers, with no learned parameters or online adaptation.

For a hand-worked step, set x = −0.5 m, v = 0.1 m/s, g = 1, d = 0.1 m/s², and ε = 0. The baseline commands u = 1 − 0.02 = 0.98 m/s², below the command limit.

Its acceleration is 1.08 m/s². Therefore x next = −0.5 + 0.02 + 0.0216 = −0.4584 m, and v next = 0.1 + 0.216 = 0.316 m/s. This sample satisfies neither success band.

Declare the evaluation distribution

Each idealized episode draws these quantities independently:

Quantity	Declared distribution
Initial position	Uniform from −1 to 0 m
Initial velocity	Uniform from −0.2 to 0.2 m/s
Actuator gain	Uniform from 0.8 to 1.2
Constant disturbance	Uniform from −0.3 to 0.3 m/s²
Position error at each action	Uniform from −η to +η m

The noise bound η defaults to 0.02 m. Gain and disturbance remain fixed within an episode; position errors change at each action. The policies receive exact velocity, so this model does not test velocity estimation, vision, delay, contact, or obstacle avoidance.

A seeded integer generator approximates these independent uniform draws. It always generates 40 position errors per episode, even when an earlier outcome stops the rollout. Increasing the episode count preserves the earlier episodes; changing the time budget preserves their conditions.

The seed makes this simulation reproducible. It supplies no hardware evidence and does not prove statistical independence. The interval calculation later assumes the declared sampling model.

For a trained policy, record its checkpoint and training seeds separately from evaluation seeds. Use a validation set for choosing checkpoints or settings, then reserve fresh evaluation episodes for the final claim. Repeatedly selecting the best evaluation seed makes that batch part of model selection; robot data leakage explains the split problem.

Separate task endings from the time budget

Our success and boundary rules define terminal task states. A timeout ends the measurement because the evaluator has waited long enough. The simulated dynamics could continue from that state.

Gymnasium's time-limit guide distinguishes this external truncation from task termination. A deadline built into a finite-horizon task can itself define termination. Which interpretation applies depends on the task definition.

Here, the budget is external and does not enter the controller observation. The implementation records success and boundary failure as terminated, and timeout as truncated. That distinction matters if a learning algorithm later uses the transitions, including its treatment of value bootstrapping.

For the reported success-by-budget metric, a timeout still contributes an unsuccessful attempt. Keep it in the denominator and report its count separately. Do not silently drop it or describe it as a proven eventual failure.

Keep each metric's denominator visible

For N attempted episodes and S successes, the empirical success rate is S/N. Each episode contributes one binary outcome, regardless of how many frames it contains.

The default 64-episode batch gives:

Controller	Successes	Boundary failures	Timeouts
Baseline	9	7	48
Damped	50	0	14

The baseline success rate is 9/64 = 14.0625%. The damped rate is 50/64 = 78.125%. Those numbers describe the declared simulator, controller settings, and six-second budget.

Mean completion time among successes uses a different denominator:

The baseline's nine successes take 13.6 seconds in total, giving 1.511111 s on average. The damped controller's 50 successes take 118.4 seconds, giving 2.368000 s. Calling the baseline better because its successful runs finish sooner would ignore its 55 unsuccessful attempts.

Report both rates and conditional times. When S = 0, the successful-completion mean is undefined; the lab displays “No successes.” A short failure duration is not a fast completion.

Compare policies on matched episodes

For each episode ID, both controllers receive the same reset, gain, disturbance, and position-error sequence. This pairing removes differences caused solely by drawing a different batch for each controller. Their states and measured positions still diverge because their actions differ.

The default outcomes form this paired count:

• Both controllers succeed: 8 episodes.
• Only the baseline succeeds: 1 episode.
• Only the damped controller succeeds: 42 episodes.
• Neither succeeds: 13 episodes.

The observed success-rate difference is (42 − 1)/64 = 64.0625 percentage points. Separate success intervals do not account for these paired disagreements. A statistical comparison should use a method that respects the paired episode outcomes.

Matched conditions improve this comparison's design, but they do not broaden its scope. Testing new objects, environments, or conditions requires a declared robot-learning generalization test. Repeated trials on one physical setup may also share failures or disturbances across episodes; that dependence changes the uncertainty analysis.

Put an interval around the success rate

A success count has sampling uncertainty under an independent, identically distributed episode model. For a fixed controller and evaluation distribution, the NIST Wilson interval gives a two-sided interval for its success probability.

Let p̂ = S/N and z = 1.959963984540054. The nominal 95% bounds are:

For the default batch, the baseline interval is [7.578573%, 24.616282%]. The damped interval is [66.567216%, 86.497681%].

The 95% label describes approximate repeated-sampling coverage under the assumptions. It is not a 95% posterior probability for this observed interval. Frames within an episode do not create extra independent trials.

The lab's seeded draws illustrate this calculation. More episodes can reduce sampling uncertainty, but they cannot repair a biased test distribution or an incorrect success detector. A handpicked checklist needs a different interpretation from random episodes drawn from a declared distribution.

Inspect the policy evaluation experiment

The first default episode shows why crossing the target is insufficient. At t = 1 s, the baseline reaches x = 0.020273 m while moving at 1.246140 m/s. It later records a boundary failure at 2.6 s.

The damped controller times out on that same episode. At 6 s, its position is 0.125754 m, outside the position band. The batch's higher damped success rate does not mean it succeeds on every condition.

Try these changes:

• Select Two-second budget. The same conditions now produce 7 baseline successes and 16 damped successes; both runs in episode 1 time out.
• Select 256 episodes. The first 64 conditions stay fixed while 192 new conditions join the batch. Counts and intervals can change together.
• Select More position noise. This changes the evaluation distribution without changing the controllers.
• Move Inspect episode. The plots change; the aggregate metrics do not.

Count complete, paired episodes

Evaluate two fixed reaching controllers

Both controllers share each reset, actuator gain, disturbance, and position-noise sequence. Both use position gain 2; velocity gains are 0.2 for the baseline and 1.8 for the damped controller. Commands saturate at ±1 m/s².

Evaluation presetCustom settingsPaired evaluationTwo-second budget256 episodesMore position noise

Episode count16 episodes64 episodes256 episodes

Evaluation seed

7

Evaluation budget (s)

6.000000

Position-noise bound (m)

0.020000

Each seed generates reproducible synthetic conditions from the distribution in the article. Every episode counts once. Changing the budget or noise changes the evaluation condition; inspecting a different episode leaves the batch metrics unchanged.

Reset policy evaluation lab

Batch outcomes and uncertainty

Baseline successes

9 of 64

Baseline boundary failures

7

Baseline timeouts

48

Damped successes

50 of 64

Damped boundary failures

0

Damped timeouts

14

Baseline success rate (%)

14.062500

Damped success rate (%)

78.125000

Baseline Wilson lower (%)

7.578573

Baseline Wilson upper (%)

24.616282

Damped Wilson lower (%)

66.567216

Damped Wilson upper (%)

86.497681

Baseline mean time among successes (s)

1.511111

Damped mean time among successes (s)

2.368000

Baseline-only successes

1

Damped-only successes

42

Wilson bounds form a nominal 95% interval for each fixed controller under independent, identically distributed episode draws. The seeded generator approximates that model. These separate intervals do not test a paired improvement. Both succeed in 8 pairs; neither succeeds in 13.

Inspect episode

1 of 64

Episode

1

Initial position (m)

-0.761219

Initial velocity (m/s)

0.165397

Actuator gain

1.044997

Constant disturbance (m/s²)

0.256189

Baseline episode outcome

Boundary failure

Damped episode outcome

Timeout

Baseline episode duration (s)

2.600000

Damped episode duration (s)

6.000000

Position and task boundaries

Velocity and the settling rule

Success requires three consecutive post-action samples inside both bands. Boundary failure takes priority, then success, then timeout. A timeout remains in the success denominator.

The position plot connects sample endpoints. Continuous position inside each interval follows a quadratic curve; velocity changes linearly. Each trace ends at that controller's first recorded outcome.

Reproduce the batch in Python

This standard-library program recreates the default conditions, paired rollouts, and metrics. Its position update uses the average of the interval's initial and final velocities, which is exact for the held acceleration. The generator matches the lab's 32-bit recurrence.

from math import sqrt

DT, N, STEPS, NOISE = 0.2, 64, 30, 0.02
seed = 7

def draw():
    global seed
    seed = (1664525 * seed + 1013904223) % (2**32)
    return seed / (2**32)

conditions = []
for _ in range(N):
    x, v = -1 + draw(), -0.2 + 0.4 * draw()
    gain, disturbance = 0.8 + 0.4 * draw(), -0.3 + 0.6 * draw()
    errors = [2 * draw() - 1 for _ in range(40)]
    conditions.append((x, v, gain, disturbance, errors))

def episode(condition, damping):
    x, v, gain, disturbance, errors = condition
    settled = 0
    for k in range(STEPS):
        u = max(-1, min(1, -2 * (x + NOISE * errors[k]) - damping * v))
        acceleration = gain * u + disturbance
        new_v = v + DT * acceleration
        x += DT * (v + new_v) / 2
        v = new_v
        settled = settled + 1 if abs(x) <= 0.1 and abs(v) <= 0.15 else 0
        if abs(x) >= 1.25:
            return "failure", (k + 1) * DT
        if settled >= 3:
            return "success", (k + 1) * DT
    return "timeout", STEPS * DT

def wilson(successes):
    z, p = 1.959963984540054, successes / N
    denominator = 1 + z*z / N
    center = (p + z*z / (2*N)) / denominator
    half = z * sqrt(p*(1-p)/N + z*z/(4*N*N)) / denominator
    return center - half, center + half

baseline = [episode(c, 0.2) for c in conditions]
damped = [episode(c, 1.8) for c in conditions]
for name, rows in [("baseline", baseline), ("damped", damped)]:
    counts = [sum(status == key for status, _ in rows)
              for key in ["success", "failure", "timeout"]]
    mean = sum(t for status, t in rows if status == "success") / counts[0]
    low, high = wilson(counts[0])
    print(f"{name}: success={counts[0]}, failure={counts[1]}, timeout={counts[2]}")
    print(f"  success={100*counts[0]/N:.6f}%, mean successful time={mean:.6f} s")
    print(f"  Wilson 95%: [{100*low:.6f}%, {100*high:.6f}%]")
base_only = sum(b[0] == "success" and d[0] != "success"
                for b, d in zip(baseline, damped))
damped_only = sum(d[0] == "success" and b[0] != "success"
                  for b, d in zip(baseline, damped))
print(f"paired: baseline-only={base_only}, damped-only={damped_only}")

Expected output:

baseline: success=9, failure=7, timeout=48
  success=14.062500%, mean successful time=1.511111 s
  Wilson 95%: [7.578573%, 24.616282%]
damped: success=50, failure=0, timeout=14
  success=78.125000%, mean successful time=2.368000 s
  Wilson 95%: [66.567216%, 86.497681%]
paired: baseline-only=1, damped-only=42

The demonstration prints a successful-completion mean for this batch, where both success counts are positive. General evaluation code must handle zero successes explicitly, as the lab does.

Evaluate the learned policy from the shared exercise

The earlier lab compares two fixed controllers. The downloadable behavior-cloning exercise adds a learned controller, fitted on separate demonstrations and selected by validation action error. It reuses the acceleration model and stopping rules explained here, but supplies a different dataset, reset range, and evaluation batch. Its numbers do not reproduce the earlier widget.

After running the package's generate, split, and train commands, run:

python3 bc.py evaluate --out run

Evaluation loads the saved model.json, checks its data and split references, and runs it without fitting again. The episode-split section explains what the model was allowed to learn from.

Declare nominal and changed dynamics

The final split supplies 16 initial states, drawn from the same x ∈ [−0.8, 0.8] m and v ∈ [−0.15, 0.15] m/s ranges as the demonstrations. Both controllers observe exact position and velocity. The learned controller is the selected affine policy; the expert reference uses clip(−2x − 1.8v, −1, 1). The expert is a comparison rule, not a guarantee of optimal control.

For each initial state, evaluate two conditions:

• Nominal: actuator gain g = 1 and constant disturbance d = 0, matching collection.
• Changed dynamics: gain drawn uniformly from [0.6, 1.4] and disturbance from [−0.25, 0.25] m/s², held constant within that rollout.

The learned policy and expert receive the same start, gain, and disturbance within each pair. Nominal and changed-dynamics batches also share their starts, so their 32 conditions are not 32 independent reset draws. The saved conditions make each comparison reproducible.

Every action uses the current simulated state. The evaluator advances the plant with that controller's action; it never substitutes the next observation from an expert demonstration. A rollout succeeds after three consecutive post-action samples satisfy |x| ≤ 0.1 m and |v| ≤ 0.15 m/s. A sampled |x| ≥ 1.25 m is a boundary failure. Evaluation checks boundary failure, then success, then the 30-action limit, which is a six-second timeout. The sampled boundary rule is not a continuous-path safety guarantee.

Inspect counts before drawing a conclusion

Each row reports 16 attempts. Success is shown as successes/attempts; boundary failures and timeouts are counts.

Trial	Success	Boundary	Timeout
Nominal, learned	16/16	0	0
Nominal, expert	16/16	0	0
Changed dynamics, learned	14/16	0	2
Changed dynamics, expert	15/16	0	1

Under nominal conditions, the mean completion time among successes is 2.225 seconds for the learned policy and 2.125 seconds for the expert. Under changed dynamics, both succeed in 14 pairs, only the expert succeeds in one, and neither succeeds in one. Counting paired outcomes shows where the difference occurs instead of treating two success rates as unrelated samples.

The frozen policy's offline action MSE, 0.00044231 (m/s²)², is computed at expert states from nominal demonstrations. It does not measure the shifted dynamics or the states produced by the learner's own actions. The two timeouts are therefore relevant findings even though the offline prediction error is small.

Inspect the evaluation summary, per-episode outcomes, and step-by-step rollouts. The last file preserves separate terminated and truncated flags: reaching the goal or boundary terminates the task; exhausting its budget truncates it. action-errors.csv in the package records the separate offline calculation.

Use the failures as the next question

Find a changed-dynamics timeout in rollout-results.csv, then inspect its final position, velocity, and preceding trace. Did it remain just outside the goal tolerance, keep moving too quickly, or fail to hold the goal for three samples? For example, learned-policy episode-016 starts its final qualifying streak at sample 29. Sample 30 is still inside, but its streak is only two, so it times out. The paired expert reaches a three-sample streak at sample 28, or 5.6 seconds. Keep that diagnosis separate from the choice of a replacement policy.

This small, deterministic batch was inspected during authoring. Its final labels are withheld by the fitting code, but it is not an untouched benchmark. Once you use these failures to choose a model, new final evidence is needed for that choice. The generalization lesson helps define the next conditions; domain randomization explains one way to vary training conditions.

Check two evaluation decisions

Exercise 1. A policy attempts 20 episodes. Twelve succeed, three record boundary failures, and five time out. Successful runs take 36 seconds in total. What success rate and conditional mean time should you report? Why would excluding timeouts change the question?

Check the episode denominators

The success rate is 12/20 = 60%. Mean completion time among successes is 36/12 = 3 s.

Excluding the five timeouts gives 12/15 = 80%, the success fraction among runs that reached a recorded task ending. It removes five attempted episodes and no longer measures success within the evaluation budget. Keep the counts 12 success, 3 boundary failure, and 5 timeout beside the rate.

Exercise 2. The final three allowed samples are (x, v) = (0.08 m, 0.10 m/s), (0.07 m, 0.16 m/s), and (0.06 m, 0.10 m/s). Earlier samples had no qualifying streak. The budget expires at the third sample. What outcome should the evaluator record? What if the middle velocity were 0.14 m/s?

Check the stopping-rule priority

With middle velocity 0.16 m/s, the consecutive counts are 1, 0, 1. The final sample records Timeout, so truncated is true and terminated is false.

With middle velocity 0.14 m/s, the counts are 1, 2, 3. Success takes priority over the coincident budget limit: record Success, with terminated true and truncated false. None of these samples meets the boundary-failure rule.

Sources and the next evaluation question

• robomimic: evaluating rollouts and saving videos documents episode count, horizon, and termination-on-success configuration. This lesson's simulator and counts are original teaching examples.
• Gymnasium: handling time limits defines termination and truncation and explains their different role in learning targets.
• NIST/SEMATECH: confidence intervals for a proportion gives the Wilson formula used in the lab.

Choose the conditions that your policy must handle next. The robot-learning generalization lesson separates new combinations of familiar conditions from conditions outside the training range.