Open-loop vs closed-loop simulation: what replay can test

A robot policy can produce sensible commands on a recording and miss its target when it drives. Its own actions change the next observation. A recording continues along the path that produced the log.

This lesson runs one policy through both tests. You will trace the point where their inputs separate, calculate the resulting scores, and identify what each score supports.

Ask what produces the next observation

In open-loop evaluation, the test supplies recorded observations while the candidate proposes actions. Those actions do not affect the next observation in the recording. This lesson calls that arrangement shadow replay.

In closed-loop simulation, the candidate acts on a simulated environment. The simulator advances its state, generates the next observation, and returns that observation to the candidate.

Both tests can evaluate a feedback policy that computes actions from observations. Here, open-loop describes a break in the test's action-to-environment-to-observation path. The control lesson uses these terms to describe a controller's connection to its plant; specify which connection you mean.

Caesar and colleagues' nuPlan paper, Section 4.2, makes this evaluation distinction concrete. Its open-loop task scores planned trajectories without using them to control the vehicle. Its closed-loop task drives the simulated vehicle and feeds the resulting state into later planning.

Compare one policy in two test harnesses

Open-loop vs closed-loop simulation

Change what the next observation depends on

The candidate rule is u = −Kx + b in both tests. Shadow replay reads a fixed expert log. The interactive simulation executes candidate commands and measures its new position.

Simulation presetCustom settingsA biased candidateMatch the recorded expertAlternate around the targetAn uninformative zero reset

Candidate gain K (1/s)

0.50

Candidate bias b (m/s)

0.10

Shared reset x₀ (m)-101

Command index

4 of 7

Next commandReset simulation comparison

Eight one-second commands, exact position observations, no noise, no other actors, and unlimited ideal velocity. The expert always uses u = −0.5x. Changing the reset creates a new expert log; changing the candidate gain or bias leaves that log fixed. The probe selects a command in the completed runs.

Recorded and interactive physical tracks

Proposed and executed commands

Recorded observation

0.062500 m

Interactive observation

0.250000 m

Recorded expert command

-0.031250 m/s

Shadow command (not executed)

0.068750 m/s

Executed candidate command

-0.025000 m/s

Replay action MSE

0.010000 (m/s)²

Recorded final position

0.003906 m

Interactive final position

0.203125 m

Interactive final check

Outside ±0.05 m

Shadow task outcome

Not measured

The final check asks only whether |x| ≤ 0.05 m at t = 8 s. It imposes no path, speed, or comfort requirement. Replay action MSE averages eight differences from expert commands on recorded observations; it measures no candidate task outcome. Readouts round to six decimal places.

Start with A biased candidate. At command 4, shadow replay proposes a positive velocity while the interactive run commands a negative velocity. The code for the policy matches across both tests; their input positions differ.

The position chart shows the recorded physical track and the interactive candidate track. Shadow replay has no candidate physical track because it never executes the proposed commands. The command chart shows those unexecuted shadow outputs separately.

Use the command index to inspect each decision. Both charts display complete runs and rescale their vertical axes as settings change. A probe change selects a sample; it does not rerun time through an animation.

Specify the robot, recording, and task

Consider a point robot moving on one line toward x = 0. Position x uses meters, velocity command u uses meters per second, and each command lasts Δt = 1 second.

This is an exact model of constant velocity over each interval. The teaching robot has exact position sensing, unlimited velocity, and no acceleration dynamics, noise, obstacles, or other actors. It executes eight commands, so the final observation occurs at t = 8 s.

An expert generates the recording with uᴱ = −0.5x, where the gain has units of inverse seconds. From x₀ = 1 m, its positions halve each second: 1, 0.5, 0.25, and so on. The log holds nine positions and eight commands.

The candidate uses u = −Kx + b. K has units of inverse seconds, and b is a velocity bias. These are supplied policy parameters; the lab performs no training.

The final check asks whether |x₈| ≤ 0.05 m. It includes no path, speed, oscillation, or comfort requirement. This narrow check provides a concrete outcome, not a complete robot task specification.

Score proposed actions on a fixed recording

Let xᴱₖ and uᴱₖ denote the expert's recorded position and command. Shadow replay computes a candidate output at each recorded position:

The MSE has units of (m/s)². It answers how closely the candidate matches eight expert actions on those recorded inputs. It supplies no candidate endpoint or task outcome.

Changing K or b leaves the recording fixed. Changing the shared reset creates a new expert recording, then starts the interactive candidate at that same position. The two tests use identical resets and zero noise.

This distinction matters in behavior cloning: action prediction on held-out demonstrations and executed policy performance measure different quantities. Keep both quantities attached to their actual data and protocol.

Execute actions to produce new observations

Let xᶜₖ denote the candidate's interactive position. Each step computes uᶜₖ = −Kxᶜₖ + b, executes that velocity for one second, and observes the resulting position.

With r = 1 − Δt K, the exact finite-step solution is:

At k = 0 the sum is empty and equals zero. This form also handles K = 0: position advances by Δt b each step. When K is positive, the equilibrium is b/K; convergence to that equilibrium requires |1 − Δt K| less than 1.

The interactive run can expose observations absent from the recording. The covariate-shift lesson studies that effect for a fitted policy. Here the policy parameters stay fixed so the experiment isolates the test harness.

Follow the first mismatch to the endpoint

Use the default K = 0.5 s⁻¹, b = 0.1 m/s, and x₀ = 1 m. The expert's first command is −0.5 m/s; the candidate's first command is −0.4 m/s. Both tests give the candidate that same initial position and output.

After one second, the recording says 0.5 m. The interactive candidate reaches 0.6 m. Shadow replay therefore proposes −0.15 m/s next, while the interactive policy commands −0.2 m/s.

Command index	Recorded x (m)	Interactive x (m)	Shadow u (m/s)	Executed u (m/s)
0	1	1	−0.4	−0.4
1	0.5	0.6	−0.15	−0.2
4	0.0625	0.25	0.06875	−0.025

On every recorded position, the candidate differs from the expert by exactly 0.1 m/s. Replay MSE is therefore 0.01 (m/s)². The interactive recurrence gives xᶜₖ = 0.2 + 0.8(0.5)ᵏ m.

At eight seconds, the recorded expert reaches 0.00390625 m, while the candidate reaches 0.203125 m. The candidate misses the stated tolerance. Its stable equilibrium is 0.2 m, so additional time alone will not remove this bias under the given model.

Check agreement and different successful actions

Match the recorded expert sets K = 0.5 and b = 0. With identical resets and exact deterministic dynamics, both runs then match at every step. Replay MSE is zero and the candidate passes the final check.

Alternate around the target sets K = 1.5 and b = 0. The candidate follows xᶜₖ = (−0.5)ᵏx₀, so it alternates sides while shrinking its error. Its replay MSE is about 0.166664 (m/s)², yet its final position at eight seconds equals the expert's final position.

That example has higher action error than the biased candidate and passes the narrow endpoint check. It establishes no general ranking between replay and rollout scores. Adding a speed limit or an oscillation penalty would change the task being scored.

An uninformative zero reset keeps K = 1.5 but starts at x₀ = 0 with b = 0. Both policies produce only zero commands. Agreement on that log cannot distinguish their behavior away from zero.

Use robot log replay for the questions it can answer

Recorded sensor streams support repeatable component comparisons. A perception system can process the same images; an estimator can process the same timed measurements; a policy can propose actions on the same logged observations. Each comparison holds the input record fixed.

For estimator testing, define initialization, timestamps, frames, and any recorded control inputs. Check estimates against available reference measurements, as in trajectory error metrics. That result covers estimation on the supplied sensor history.

The official ROS 2 Jazzy recording tutorial shows recording topics and replaying messages. Its example includes velocity commands and turtle poses. Replaying command messages into a simulator can move its robot, so the word “replay” alone does not identify the evaluation wiring.

A shadow test must keep candidate outputs out of the environment that supplies its observations. If a test moves a candidate robot while continuing to feed its old recorded sensor views, describe that hybrid explicitly. A changed viewpoint may require newly rendered observations or another justified sensor model.

Offline policy-value estimation can also use logged data under explicit assumptions. Jiang and Li's off-policy evaluation paper studies that separate estimation problem. Matching logged action labels alone does not implement such an estimator or establish its assumptions.

State which parts of the environment react

Closing the robot's loop says that its actions affect its later state and observations. It does not guarantee that every other actor responds. nuPlan's original task definitions distinguish closed-loop tests with recorded other-agent trajectories from tests with models for reactive agents.

This lab has no other actors and observes position directly. It cannot test sensor rendering, contact, latency, human interaction, or hardware transfer. A real simulation needs credible models for the features your claim depends on. The robotics simulation overview separates mechanics, numerical integration, sensing, and timing so you can identify which part supports each claim.

Write down a test contract before reporting a score:

• Which inputs come from logs, and which observations reflect candidate actions?
• What are the reset distribution, time step, dynamics, and sensing assumptions?
• Which commands actually execute, and which outputs only enter a metric?
• What defines completion, failure, and the time limit?
• Which constraints and interactions does the test omit?

Use robot policy evaluation to turn that contract into episode metrics. A single deterministic example explains the mechanism; it estimates no real-world success rate.

Reproduce both evaluations in Python

This standard-library example generates the expert recording and executes the candidate recurrence. The separate closed-form assertion checks the interactive endpoint without repeating its step loop.

def compare(gain, bias, start):
    recorded = [start * 0.5 ** k for k in range(9)]
    expert = [-0.5 * x for x in recorded[:-1]]
    shadow = [-gain * x + bias for x in recorded[:-1]]
    positions, actions = [start], []
    for _ in range(8):
        action = -gain * positions[-1] + bias
        actions.append(action)
        positions.append(positions[-1] + action)  # dt = 1 second
    mse = sum((u - v) ** 2 for u, v in zip(shadow, expert)) / 8
    r = 1 - gain
    endpoint = r ** 8 * start + bias * sum(r ** j for j in range(8))
    assert abs(endpoint - positions[-1]) < 1e-12
    return recorded, shadow, positions, actions, mse


recorded, shadow, positions, actions, mse = compare(0.5, 0.1, 1.0)
print(f"default replay MSE: {mse:.6f} (m/s)^2")
print(f"default x4: log={recorded[4]:.6f}, interactive={positions[4]:.6f} m")
print(f"default u4: shadow={shadow[4]:.6f}, executed={actions[4]:.6f} m/s")
print(f"default x8: log={recorded[-1]:.6f}, interactive={positions[-1]:.6f} m")
for name, gain, bias, start in (
    ("matched", 0.5, 0.0, 1.0),
    ("alternating", 1.5, 0.0, 1.0),
    ("zero reset", 1.5, 0.0, 0.0),
):
    _, _, positions, _, mse = compare(gain, bias, start)
    passed = abs(positions[-1]) <= 0.05
    print(f"{name}: MSE={mse:.6f}, x8={positions[-1]:.6f}, inside={passed}")

Output:

default replay MSE: 0.010000 (m/s)^2
default x4: log=0.062500, interactive=0.250000 m
default u4: shadow=0.068750, executed=-0.025000 m/s
default x8: log=0.003906, interactive=0.203125 m
matched: MSE=0.000000, x8=0.003906, inside=True
alternating: MSE=0.166664, x8=0.003906, inside=True
zero reset: MSE=0.000000, x8=0.000000, inside=True

Try it yourself

Exercise 1. Keep K = 0.5, set b = −0.1 m/s, and start at 1 m. Calculate the second observed position x₁ in both tests, then the candidate's next command in each. What are replay MSE and the interactive endpoint at eight seconds?

Show solution: reverse the bias

The recording still gives xᴱ₁ = 0.5 m. The candidate first commands −0.6 m/s, so xᶜ₁ = 0.4 m. At command 1, shadow replay proposes −0.35 m/s and the interactive policy executes −0.3 m/s.

Every recorded action error is −0.1 m/s, giving MSE = 0.01 (m/s)². The interactive path is xᶜₖ = −0.2 + 1.2(0.5)ᵏ m, so xᶜ₈ = −0.1953125 m, outside the final tolerance.

Exercise 2. The zero-reset preset produces zero action error and identical paths. Change only the shared reset to 1 m. At command 1, calculate the recorded and interactive observations and the two candidate commands. Does agreement at zero establish agreement on the new run?

Show solution: test a revealing reset

The candidate still uses K = 1.5 and b = 0. From 1 m it commands −1.5 m/s, reaching −0.5 m at t = 1 s. The expert recording reaches 0.5 m.

Shadow replay proposes −0.75 m/s on the recorded observation. The interactive candidate executes 0.75 m/s on its own observation. Agreement at the zero reset supplied no evidence about this difference away from zero.

Sources and further study

• Caesar et al., nuPlan, original benchmark paper (2021, revised 2022). Section 4.2 defines open-loop, non-reactive closed-loop, and reactive closed-loop tasks.
• ROS 2 Jazzy documentation, Recording and playing back data. The official documentation source explains topic recording and message replay, including command topics.
• Jiang and Li, Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2016). This paper studies value estimation from another policy's data, a different procedure from the shadow action score used here.

Next, define the resets and outcomes your own comparison needs with robot policy evaluation. Keep the log-input score and the executed task result separate in the report.