Behavior cloning covariate shift: when actions change the inputs

Follow the feedback behind covariate shift

A policy's action changes the input to its next prediction. A small action error can move a robot into states that its expert demonstrations rarely visited. The next prediction then faces a different input distribution.

In behavior cloning, a learner fits expert observation-action pairs. An offline test asks how well it predicts actions on supplied observations. A rollout lets its own actions determine the observations that follow. The simulation-loop comparison makes that boundary explicit by feeding the same policy recorded states in one test and its own resulting states in another.

Ross, Gordon, and Bagnell's DAgger paper studies this dependence between actions and future inputs. Its analysis motivates evaluating a policy under the states it induces. This lesson makes that feedback visible with a deterministic motion model, fitted policies, and matched starts.

Fix the dynamics and the expert rule

Consider the lateral motion of a point robot. Its full observed state is s = (y, v): lateral position y in meters and lateral velocity v in meters per second. The target is rest at y = 0.

An action u commands acceleration in m/s², held constant for Δt = 1 s. Integrating that constant acceleration gives exact sample endpoints:

The expert applies this fixed feedback rule:

The position coefficient has units 1/s²; the velocity coefficient has units 1/s. The velocity term brakes motion as the robot approaches the target. We use exact state measurements, unlimited acceleration, and no disturbances, obstacles, or delays.

Both expert and learner use these same dynamics throughout. This differs from the changed actuator model in sim-to-real transfer. Here, the policy's actions create the state-distribution change.

Separate training episodes from expert evaluation

Each training episode starts with zero velocity at one of four positions: −0.10, −0.05, +0.05, or +0.10 m. The expert takes eight actions. We record the state immediately before each action and its expert label, giving 32 training pairs.

Four separate expert episodes start at −0.075, −0.025, +0.025, and +0.075 m. Their eight actions supply 32 held-out pairs. Neither policy uses those pairs when fitting its coefficients.

For example, a training episode starting at y = 0.10 m begins:

Decision step	y (m)	v (m/s)	Expert u⋆ (m/s²)
0	0.1000	0.0000	−0.0600
1	0.0700	−0.0600	0.0300
2	0.0250	−0.0300	0.0210
3	0.0055	−0.0090	0.0075

The final state after action eight has no recorded action in that episode. It does not enter either dataset. Both datasets stay fixed when you change the rollout start or horizon.

These hand-chosen episodes illustrate evaluation populations. They are not an untouched benchmark or a sample supporting claims about physical robots. For recordings with shared scenes or repeated sessions, the robot-data leakage lesson explains how to choose splits that test the intended generalization.

Fit two policy classes once

Both policy classes receive the full state (y, v). The position-only fit fixes the velocity coefficient to zero by design. The position + velocity fit allows both coefficients to vary:

We fit each class by minimizing the mean squared expert-action error over the same 32 training pairs. There is no intercept or regularization. The position-only minimizer is a = Σyu⋆ / Σy²; the two-coefficient fit solves the corresponding two-column least-squares problem.

Policy class	Fitted a (1/s²)	Fitted b (1/s)	Training action RMSE (m/s²)
Position-only	−0.214286	0.000000	0.018298
Position + velocity	−0.600000	−1.200000	0.000000

The second class contains the exact expert rule. The training states identify both coefficients, so the fit recovers it up to floating-point roundoff. The restricted class cannot express the expert's velocity response.

After fitting, coefficients remain fixed during every rollout. Selecting a policy class chooses its fitted rule. Changing the rollout start, horizon, or displayed step never retrains it.

Calculate the first departure from the expert

Start both rollouts at y₀ = 0.075 m and v₀ = 0. The expert requests −0.045 m/s². The position-only learner requests approximately −0.016071 m/s².

After one second:

• Expert: y₁ = 0.075 + ½(−0.045) = 0.052500 m; v₁ = −0.045000 m/s.
• Learner: y₁ = 0.075 + ½(−0.01607143) = 0.066964 m; v₁ = −0.016071 m/s.

They started together and used the same mechanics. Their first different actions created different next states. At later steps, each policy acts on its own current state.

The restricted policy also lacks damping in this example. With coefficient a and a one-second hold, its state-update matrix is:

For the fitted a, det(A) = 1 − a/2 ≈ 1.107143. Its eigenvalues form a complex pair with magnitude √det(A) ≈ 1.052209, above one. This sampled controller produces growing oscillations from a nonzero start; that is a property of this fitted model, not a universal behavior-cloning growth law.

Measure action error on two state populations

The held-out score evaluates the fixed learner on 32 states produced by the expert. For each state sᵢ, it compares π(sᵢ) with the expert label u⋆(sᵢ):

The position-only fit scores 0.012939 m/s² on those held-out states. Its training RMSE is higher because the training starts have larger mean squared offsets. The difference follows from the chosen data magnitudes; it does not establish better performance on unseen robots.

The learner-state action RMSE uses the learner's own decision states. At each one, the evaluator asks what the unchanged expert rule would command there. It does not subtract the action from the expert's separate rollout, which may be at a different state.

For the default start, the learner-state RMSE is 0.046357 m/s² over the first eight actions. It reaches 0.097751 m/s² over 32 actions. The held-out expert-state score stays 0.012939 m/s² in both cases.

These scores summarize different state populations: four fixed expert episodes versus one learner rollout prefix. They are reproducible diagnostics, not interchangeable estimates of one test distribution. Position and velocity traces answer a separate question: what motion did those actions produce?

Inspect the states a policy visits

For a displayed prefix of k actions, the learner's empirical visitation distribution assigns equal weight 1/k to states s₀ through sₖ₋₁. It excludes the final state sₖ because the prefix has not acted there. At k = 0, no learner-state action score exists yet.

The state-space plot places each decision at its position and velocity coordinates. Repeated states overlap, so a scatter plot alone does not display every frequency clearly. The counts and population definitions matter alongside the marks.

Training states lie within |y| ≤ 0.10 m and |v| ≤ 0.06 m/s. The plotted rectangle records only these coordinate bounds. A point inside it can still lie far from every demonstration; the box is not a probability region or a coverage guarantee.

In the default 32-action learner rollout, 21 decision states lie outside that rectangle. Its final position is −0.167602 m and its velocity is −0.168251 m/s. The matched-start expert has nearly reached rest.

That trajectory begins at a held-out start inside the training coordinate range. The learner reaches unfamiliar combinations of position and velocity through its own earlier actions.

Distinguish covariate shift from changed labels

In the standard supervised-learning definition, covariate shift changes the input distribution while preserving the conditional label distribution given that input. Sugiyama, Krauledat, and Müller's paper states this distinction explicitly.

Here, the input is the full state (y, v). Its visitation distribution changes, while u⋆(y, v) stays the same deterministic rule everywhere. We use “concept shift” for a changed conditional label rule at the same stated input, such as a different expert objective requiring a different action for an otherwise identical full state.

The input definition matters. If you observe only y, different hidden velocities can require different expert actions at the same observed position. A changed mixture of those velocities can change the conditional action distribution given y, even when the full-state expert rule stays fixed.

Our position-only fit receives both inputs and constrains one coefficient to zero. That modeling restriction causes prediction errors; rollout feedback changes where those errors occur. Neither fact requires a change in the simulator or expert.

Compare frozen policies in the lab

A frozen policy changes its own inputs

Compare expert-state loss with a policy rollout

Both policies receive position and velocity. The position-only fit constrains the velocity coefficient to zero. Each policy learns from the same 32 expert examples, then keeps its coefficients fixed.

Covariate-shift presetCustom settingsPosition-only fitPosition + velocity fitShort rolloutStart at rest on target

Policy classPosition-only fitPosition + velocity fit

Shared initial offset (m)

0.075000

Rollout horizon (steps)

32

Both rollouts start at the same position with zero velocity. Dynamics, expert labels, and the 32 held-out expert examples stay fixed. Each action holds a lateral acceleration for one second. Settings show a complete new rollout; replay changes only the displayed prefix.

Displayed rollout step

32 of 32

Restart rolloutStep rolloutFinish rolloutReset covariate-shift lab

Lateral position along each rollout

Step

32

Training action RMSE (m/s²)

0.018298

Held-out expert-state action RMSE (m/s²)

0.012939

Learner-state action RMSE (m/s²)

0.097751

Learner lateral position (m)

-0.167602

Expert lateral position (m)

0.000000

Learner lateral velocity (m/s)

-0.168251

Fitted position coefficient (1/s²)

-0.214286

Fitted velocity coefficient (1/s)

0.000000

Decision states outside demo box

21 of 32

The learner chooses later actions on states created by its earlier actions. Compare those states and their action errors with the expert data.

The states where actions are chosen

Try these comparisons:

1. Position-only fit: inspect the complete rollout, then restart and take one step. Check how the first action creates a different next state.
2. Short rollout: compare eight actions with 32. The expert evaluation dataset and fitted coefficients stay unchanged.
3. Position + velocity fit: both rollouts agree because this class represents the expert rule. Behavior cloning succeeds in this exact setting.
4. Start at rest on target: the restricted policy also stays still from y = v = 0. This start does not expose its missing velocity response.

Changing a setting shows the complete new rollout. Replay reveals prefixes without modifying the policy. Reset restores the position-only fit, start 0.075 m, and 32-step horizon.

More varied training can expose missing recovery behavior. A class constrained to ignore velocity still cannot represent this expert's full rule. DAgger addresses learner-state coverage by collecting expert labels at visited states and aggregating them for training; model capacity and observation quality remain relevant.

Reproduce the fit and rollout in Python

This program needs only the Python standard library. It creates separate expert episodes, fits both classes, then evaluates frozen coefficients. The expert-state test always contains the same 32 rows.

from math import sqrt

def expert(y, v):
    return -.6*y - 1.2*v

def advance(y, v, u):
    return y + v + .5*u, v + u  # one-second acceleration hold

def demonstrations(starts):
    rows = []
    for y in starts:
        v = 0.0
        for _ in range(8):
            u = expert(y, v)
            rows.append((y, v, u))
            y, v = advance(y, v, u)
    return rows

train = demonstrations([-.1, -.05, .05, .1])
held = demonstrations([-.075, -.025, .025, .075])
yy = sum(y*y for y, v, u in train)
yv = sum(y*v for y, v, u in train)
vv = sum(v*v for y, v, u in train)
yu = sum(y*u for y, v, u in train)
vu = sum(v*u for y, v, u in train)
det = yy*vv - yv*yv
policies = {"position": (yu/yy, 0.0),
            "state": ((yu*vv-vu*yv)/det, (vu*yy-yu*yv)/det)}

def rmse(policy, rows):
    a, b = policy
    return sqrt(sum((a*y+b*v-u)**2 for y, v, u in rows) / len(rows))

def rollout(policy, horizon, start=.075):
    a, b = policy
    y, v = start, 0.0
    squared_error = 0.0
    outside = 0
    for _ in range(horizon):
        u = a*y + b*v
        squared_error += (u - expert(y, v))**2
        outside += abs(y) > .1 + 1e-12 or abs(v) > .06 + 1e-12
        y, v = advance(y, v, u)
    return y, v, sqrt(squared_error/horizon), outside

a, b = policies["position"]
print(f"position coefficient: {a:.9f}")
print(f"held-out action RMSE: {rmse(policies['position'], held):.6f}")
for horizon in [8, 32]:
    y, v, error, outside = rollout(policies["position"], horizon)
    print(f"{horizon} steps: y={y:.6f}, v={v:.6f}, "
          f"own-state RMSE={error:.6f}, outside={outside}")
a, b = policies["state"]
print(f"state coefficients: {a:.6f}, {b:.6f}")
_, _, error, outside = rollout(policies["state"], 32)
print(f"state fit: own-state RMSE={error:.6f}, outside={outside}")

Expected output:

position coefficient: -0.214285738
held-out action RMSE: 0.012939
8 steps: y=-0.094121, v=0.024082, own-state RMSE=0.046357, outside=1
32 steps: y=-0.167602, v=-0.168251, own-state RMSE=0.097751, outside=21
state coefficients: -0.600000, -1.200000
state fit: own-state RMSE=0.000000, outside=0

The action RMSE unit is m/s²; its squared loss has units m²/s⁴. The outside count uses pre-action states and a small numerical boundary tolerance. It does not label every unseen state or quantify a probability of failure.

Check two rollout calculations

Exercise 1: At a learner state y = 0.05 m and v = −0.02 m/s, use a = −0.214286 and b = 0. What actions do the learner and expert request? Where does the learner end after one second?

Check the action and next state

The learner requests a y = −0.0107143 m/s². The expert requests −0.6(0.05) − 1.2(−0.02) = −0.006 m/s². The learner reaches y′ = 0.05 − 0.02 + ½(−0.0107143) = 0.02464285 m and v′ = −0.02 − 0.0107143 = −0.0307143 m/s. Both action predictions use the same learner state.

Exercise 2: A three-action prefix visits decision states s₀, s₁, and s₂, then ends at s₃. Its action-error magnitudes are 0.01, 0.02, and 0.03 m/s². A hypothetical next action at s₃ would have error 0.20 m/s². Which errors belong in the displayed prefix RMSE, and what is the result?

Check the evaluation population

Use the three decisions already taken. Their RMSE is √[(0.01² + 0.02² + 0.03²)/3] = 0.021602 m/s². The final state s₃ contributes only after a fourth action enters the displayed prefix. Its hypothetical error does not change the three-action score.

Sources and further study

• Ross, Gordon, and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2011: policy-induced state distributions, sequential prediction errors, and dataset aggregation. The paper's bounds depend on its stated assumptions; this lab does not estimate them.
• Sugiyama, Krauledat, and Müller, Covariate Shift Adaptation by Importance Weighted Cross Validation, 2007: the distinction between a changed input distribution and an unchanged conditional output distribution.
• Russ Tedrake, MIT Underactuated Robotics: Imitation Learning: behavior cloning as supervised policy fitting and the distinction between full-state and observation-based policies.

Continue with DAgger imitation learning to inspect the expert queries and dataset updates needed to learn from learner-visited states.