Behavior cloning from observation: infer missing actions

Behavior cloning from observation learns a policy from expert motion without the expert’s action labels. A learned inverse model can supply those missing labels. Errors in that model then enter the policy’s training data.

Here you will fit that inverse, infer actions, and execute a cloned policy. A small deterministic experiment exposes both missing exploration and misleading sensor readings. It uses supplied scalar states, with no video model or hardware result.

Define behavior cloning from observation

An expert recording might show where a robot moved while omitting its motor commands. Ordinary behavior cloning expects observation-action examples. Imitation from observation names the broader setting where demonstrations omit actions.

Behavioral cloning from observation (BCO) is a specific approach within that setting. Torabi, Warnell, and Stone’s 2018 paper learns inverse dynamics from the learner’s own experience, then uses that model to infer actions for state-only demonstrations. A supervised policy fit follows.

The missing labels belong to the expert data. The learner still records which actions it executes during exploration. That distinction determines what the method can learn.

Follow the two learning stages

Keep four operations separate:

1. Execute learner actions and record transitions with their action labels.
2. Fit an inverse model that predicts an action from consecutive states.
3. Apply that inverse to the expert’s state-only transitions.
4. Fit a policy to the expert states and inferred actions.

The original paper also describes repeated learner interaction and refitting after demonstrations arrive. This lesson performs one inverse fit and one policy fit, illustrating the basic workflow. Its small least-squares models differ from the paper’s learned action distributions.

The imitation-learning map places BCO beside methods with other data requirements. Learning a model that predicts actions from transitions also differs from learning a reward that explains behavior.

Fix the state and action meanings

The teaching environment has a complete scalar state x, measured in meters. One command u contributes a displacement, also in meters, during a fixed unit interval. The exact discrete plant is:

This mathematical plant includes a fixed contraction toward zero. The command u is the actuator’s displacement contribution; total displacement is x next − x. No velocity, hidden actuator state, collision, or action saturation enters this example.

The expert uses u = −0.25x, so its next state is 0.25x. Demonstrations contain four transitions from each reset, −0.8 m and +0.8 m. Their ten recorded states supply eight consecutive-state pairs.

The inverse model and policy fit receive no expert actions. The simulator uses the expert rule to construct the demonstration, and a separate audit consults it after fitting. Evaluation executes both policies from the same selected reset for eight commands.

Collect the learner’s labeled transitions

Each exploration row resets the learner, executes one command, and reads the next state. Four resets, x = −1, −0.5, 0.5, and 1 m, each receive two commands:

The action-diversity control a has units of meters. At a = 0, both commands at a given reset coincide. Eight recorded rows then contain only four distinct transitions.

The sensing control η perturbs only the learner’s next-state reading:

This fixed sign pattern is a deterministic stress test. It is balanced across the eight rows; it does not represent sampled sensor statistics. Current learner states, expert observations, and rollout states remain exact.

For example, the default x = 1 m and v = 1 give u = 0.5 m. The true next state is 1 m. With η = 0.125 m, the learner records 1.125 m.

Fit a one-step inverse model

Write Δx = x̃ next − x for the observed learner change. Fit two dimensionless coefficients by ordinary least squares, without an intercept or regularization:

The exact physical inverse is u = Δx + 0.5x when readings are exact. The widget estimates both coefficients from data; it does not insert these known values into the fit. This is a learned one-step transition inverse, distinct from the force and torque equations in rigid-body inverse dynamics.

The balanced exploration design permits a hand-checkable solution. When a² + η² is positive:

To derive this, remove each column’s linear dependence on x. The remaining observed change is av + η sign(x)v; the remaining commanded action is av. The two sign patterns have zero inner product across the eight rows, which gives the ratio above.

At the defaults, a = 0.25 m and η = 0.125 m, βΔ = 0.8 and βx = 0.45. The inverse’s training action RMSE is 0.111803 m. Input measurement error has changed the fitted relationship.

Infer labels and fit the policy

Expert transitions have exact change Δx = −0.75x. Applying the fitted inverse gives inferred labels:

At the defaults, k = 0.45 − 0.75 × 0.8 = −0.15. For the expert transition 0.8 m → 0.2 m, the inverse supplies −0.12 m. The hidden generating action was −0.20 m, so this label has error +0.08 m.

All eight inferred labels lie on the line u = −0.15x. The policy therefore achieves zero error against its training labels. The separate audit against hidden expert commands reports RMSE 0.041312 m across those same eight transitions.

Executing the policy changes the next state. From x = 1 m, its first command is −0.15 m and its next state is 0.35 m. The expert reaches 0.25 m after that step. Policy evaluation explains why action fitting and executed behavior require separate checks.

Compare inference with executed behavior

Collect, infer, clone, then evaluate

Can a learned inverse recover the missing actions?

Eight learner transitions include commanded actions. Two expert trajectories provide states only. Fit an inverse model, infer eight expert labels, fit u = kx, and execute the frozen policy in x next = 0.5x + u.

Observation cloning presetCustom experimentPerturbed readingsClean diverse actionsNo action diversityApparent rank from noise

Exploration action diversity

0.250000 m

Next-state perturbation

0.125000 m

Learner commands are 0.25x ± a. Only their next-state readings receive the deterministic perturbation ±η. Expert observations and evaluation states are exact. This is a controlled sensing stress test, without images or sampled sensor statistics.

Reset observation cloning lab

Inverse fit status

Unique least-squares fit

State coefficient

0.450000

State-change coefficient

0.800000

Inverse training RMSE (m)

0.111803

Inferred-label audit RMSE (m)

0.041312

Policy gain

-0.150000

Policy fitting RMSE (m)

0.000000

Learned final position (m)

0.000225

Reference final position (m)

0.000015

Rollout trajectory RMSE (m)

0.040121

The perturbed learner inputs produce biased inferred labels. A zero policy fitting error only means the policy matches those inferred labels.

Inspect learner transition

8 of 8

Which transitions support the inverse?

Learner state (m)

1.000000

Recorded learner action (m)

0.500000

True learner next state (m)

1.000000

Observed learner next state (m)

1.125000

Rollout start position

1.000000 m

Inspect rollout step

4 of 8

Execute the inferred policy on its own states

Inspected reference position (m)

0.003906

Inspected learned position (m)

0.015006

Policy fitting RMSE scores the eight inferred labels. The separate label audit uses hidden synthetic expert actions only after fitting. Neither score measures a population success rate. At the zero reset, every fitted policy shown stays at zero, so that rollout cannot distinguish their quality.

Start with Perturbed readings, then choose Clean diverse actions. Exact, diverse learner data recover βx = 0.5 and βΔ = 1. The policy becomes k = −0.25 and matches the expert under this plant.

The coverage plot compares learner and expert transitions. It shows which combinations of state and state change the inverse sees. The rollout plot executes each policy on its own resulting states, using the same reset.

Read the metrics with their populations:

• Inverse training RMSE: action prediction error on eight learner transitions.
• Inferred-label audit RMSE: error against hidden expert actions on eight demonstration transitions, used only after fitting.
• Policy fitting RMSE: error against the eight inferred labels.
• Rollout trajectory RMSE: position difference across nine matched samples, including the shared reset and terminal sample.

At the default reset of 1 m, learned and reference final positions are 0.000225 m and 0.000015 m. Both approach zero in this example, but their sampled trajectories differ. The trajectory RMSE is 0.040121 m; it is not a success probability or hardware estimate.

Training always uses the same expert resets. Changing the rollout start or inspection step changes no fitted coefficient. The default test reset differs from the expert resets, while also appearing among learner exploration resets.

Diagnose missing and misleading coverage

With No action diversity, a = η = 0. Exploration obeys u = 0.25x and Δx = −0.25x. These rows identify only βx − 0.25βΔ = 0.25; infinitely many coefficient pairs satisfy that condition.

The widget reports Rank deficient and stops action inference. It keeps the expert reference visible. A small residual could not choose the correct inverse from that family.

Now select Apparent rank from noise. Action diversity remains zero, but η is positive. The measured transition matrix has full column rank, and the unique fit is βx = 0.25, βΔ = 0.

That fit exactly predicts the exploration commands using x alone. It achieves zero training error while ignoring state change. Its inferred policy is u = +0.25x, giving rollout multiplier 0.75. The expert’s multiplier is 0.25.

Numerical rank describes the recorded matrix. Causal identification also depends on the experiment and measurement process. These eight structured readings cannot support a calibrated uncertainty interval or a claim about all exploration strategies.

Check what observation leaves out

The experiment supplies a state that fully determines the next-state distribution together with the action. That is a Markov state assumption. A camera observation can hide velocity, contact, or actuator conditions, so a single image need not supply such a state.

For a separate constant-acceleration example, position change over Δt satisfies Δx = vΔt + 0.5aΔt². If the starting velocity v is unknown, one observed position change can support several accelerations a. History or extra sensors may provide the missing information.

MIT’s imitation-learning notes distinguish state-feedback policies from policies using observation histories. Even with complete states, stochastic transitions or redundant actuators can leave several plausible actions. A distribution over actions can express ambiguity that a single squared-error estimate hides.

State coordinates, action units, and observation intervals must also agree. A human arm and a robot arm do not automatically share an action space or feasible motions. Different bodies can require a correspondence model and learner-specific feasibility checks before an observed transition becomes a useful training target.

This lab assumes the expert and learner share the same scalar plant. It models no image extraction or cross-body correspondence. It also performs no repeated data collection after cloning, and changing its settings to improve an audit score turns that score into model-selection evidence. Reserve separate evaluation conditions, as discussed in robot data leakage.

Reproduce the complete calculation

This standard-library example solves the two-column normal equations independently of the widget’s orthogonalized fit. It reproduces the default case and its eight-command rollout. The rank guard prevents division by a zero determinant for the unexcited, unperturbed case.

from math import sqrt

a, eta = 0.25, 0.125
rows = []
for x in (-1.0, -0.5, 0.5, 1.0):
    for sign in (-1, 1):
        u = 0.25*x + a*sign
        true_next = 0.5*x + u
        observed_next = true_next + eta*(1 if x > 0 else -1)*sign
        rows.append((x, observed_next-x, u))
xx = sum(x*x for x, d, u in rows)
xd = sum(x*d for x, d, u in rows)
dd = sum(d*d for x, d, u in rows)
xu = sum(x*u for x, d, u in rows)
du = sum(d*u for x, d, u in rows)
det = xx*dd - xd*xd
if abs(det) < 1e-12:
    raise ValueError("Inverse coefficients are not uniquely identified")
bx = (xu*dd - du*xd)/det
bd = (du*xx - xu*xd)/det
inverse_rmse = sqrt(sum((bx*x + bd*d-u)**2 for x,d,u in rows)/8)

# Expert input contains only states. Hidden commands do not enter either fit.
expert = [(start*0.25**j, start*0.25**(j+1))
          for start in (-0.8, 0.8) for j in range(4)]
inferred = [(x, bx*x + bd*(nxt-x)) for x,nxt in expert]
k = sum(x*u for x,u in inferred)/sum(x*x for x,u in inferred)
fit_rmse = sqrt(sum((k*x-u)**2 for x,u in inferred)/8)
# Consult the generating expert rule only for this post-fit audit.
audit_rmse = sqrt(sum((u-(-0.25*x))**2 for x,u in inferred)/8)
learned, reference = [1.0], [1.0]
for _ in range(8):
    learned.append(0.5*learned[-1]+k*learned[-1])
    reference.append(0.25*reference[-1])
trajectory_rmse = sqrt(sum((x-y)**2 for x,y in zip(learned,reference))/9)
print(f"inverse coefficients: {bx:.6f}, {bd:.6f}")
print(f"inverse training RMSE: {inverse_rmse:.6f} m")
print(f"inferred action at x=0.8: {inferred[4][1]:.6f} m")
print(f"policy gain: {k:.6f}; fit RMSE: {fit_rmse:.6f} m")
print(f"inferred-label audit RMSE: {audit_rmse:.6f} m")
print(f"final positions: {learned[-1]:.6f}, {reference[-1]:.6f} m")
print(f"trajectory RMSE: {trajectory_rmse:.6f} m")

Expected output:

inverse coefficients: 0.450000, 0.800000
inverse training RMSE: 0.111803 m
inferred action at x=0.8: -0.120000 m
policy gain: -0.150000; fit RMSE: 0.000000 m
inferred-label audit RMSE: 0.041312 m
final positions: 0.000225, 0.000015 m
trajectory RMSE: 0.040121 m

Try it yourself

Exercise 1. Use exact learner sensing and positive action diversity. For an expert transition from 0.8 m to 0.2 m, calculate the inferred command using βx = 0.5 and βΔ = 1. Then execute that command in the learner plant.

Check the inferred action and next state

The change is 0.2 − 0.8 = −0.6 m. The inverse gives u = 0.5 × 0.8 − 0.6 = −0.2 m. Executing it gives x next = 0.5 × 0.8 − 0.2 = 0.2 m, matching the observed transition.

Exercise 2. Use the misleading preset’s coefficients, βx = 0.25 and βΔ = 0, on that same expert transition. Calculate the inferred command, its error against the hidden −0.2 m command, and the learner’s resulting next state. Does zero exploration training error resolve this mistake?

Check what the zero training error misses

The inferred command is 0.25 × 0.8 = +0.2 m. Its error is +0.4 m. The learner reaches 0.5 × 0.8 + 0.2 = 0.6 m, which differs from the observed 0.2 m target.

The exploration commands all obeyed u = 0.25x. A model can reproduce those labels exactly while learning nothing useful from state change. Its zero training error does not establish the correct inverse.

Sources and further study

• Torabi, Warnell, and Stone, Behavioral Cloning from Observation, IJCAI 2018, especially Sections 3–4: state-only demonstrations, the inverse-model stage, action inference, and policy fitting.
• Russ Tedrake, Imitation Learning, Underactuated Robotics: supervised policy fitting and the distinction between states and observation histories.

Use the imitation-learning comparison to decide whether your available demonstrations, learner interactions, and expert access support this workflow. Before trusting inferred labels, identify which parts of the transition your sensors actually observe.