explainer
Sim-to-real transfer: train for a model that changes
Explore sim-to-real transfer by fitting a policy across actuator models. Test unseen gains, compare goal error and effort, and distinguish simulation checks from hardware evidence.
What you will learn
- Distinguish visual mismatch from mismatch in motion and actuation.
- Explain how calibration and varied training address different sources of error.
- Fit a policy by minimizing average loss over several training models.
- Evaluate fixed policies on the same unseen test setting.
- Separate a simulation stress test from evidence of transfer to hardware.
Before you start
A controller can finish a task in simulation and fail when the actuator responds differently. Sim-to-real transfer asks whether a method developed in simulation also works on physical hardware. You need to test the gap between those environments, including the parts your simulator leaves out.
This lesson fits two tiny policies and tests them on changed actuator gains. The experiment uses simulation on both sides of the comparison. Published robot experiments provide separate evidence for the broader approach.
Define sim-to-real transfer
The reality gap is the mismatch between a simulated environment and the physical system. A policy can rely on details that change at deployment: a clean image, exact friction, immediate commands, or perfect state measurements. Good simulated task scores alone leave those dependencies untested.
A transfer claim needs a clear source and destination. Record what the simulator supplied, what the model learned, and what measurements came from the physical test. A successful run supports the conditions of that test, with broader claims requiring broader evidence. The robotics simulation overview distinguishes checking an implementation against its equations from validating those equations against physical measurements.
In Tobin and colleagues' 2017 paper, the authors trained object-localization networks on randomized rendered images and tested them on real camera images. They also used the detector in a robotic grasping setup. Those experiments establish results for their system and tasks; our small scalar model does not reproduce them.
Separate visual and dynamics mismatch
Visual mismatch changes what a sensor sees. Lighting, texture, camera pose, and distractors can alter an image while the task stays the same. Tobin and colleagues varied rendered scene properties to study that form of transfer.
Dynamics mismatch changes what an action does. Mass, friction, damping, actuator response, and command timing can change the next state. Peng and colleagues' 2018 paper trained policies with varied dynamics for a robotic-arm puck-pushing task and evaluated transfer on a real Fetch arm.
The two gaps can occur together. Better image variation does not establish tolerance to an unmodeled delay. A dynamics experiment with exact position observations leaves image-based perception untested.
Measure what you can calibrate
Calibration establishes measurement and command conventions: coordinate frames, sensor scales, and actuator units. System identification estimates model parameters from observed responses. The dynamic-parameter identification lesson shows how measured motion constrains a model fit.
For example, command a known displacement and measure the resulting change in position. A consistent scale error suggests a parameter to estimate. A response that changes with direction, load, or speed suggests that a single constant gain leaves important behavior unexplained.
Calibration and varied training can work together. Use measurements to set a plausible center and range, then train over uncertainty that remains. The model-uncertainty lesson separates chosen parameter bounds from claims about every possible system.
Train across a distribution of models
Domain randomization varies aspects of the training environment. A dynamics policy can train across different masses or gains; a vision model can train across different renderings. The distribution determines which variations the training objective rewards handling well.
Peng and colleagues formulate training as expected return over a distribution of dynamics models. Their experiments vary properties including mass, damping, friction, controller gains, and action timing. The paper also tests policy architecture and individual randomization choices.
For a loss L and domain parameter g, an average-loss version is:
θ* = arg minθ E[g ∼ Ptrain][L(θ, g)]
An expected loss weights cases by their probability. A worst-case objective would minimize the largest loss over a stated set, which is a different optimization problem. Neither objective automatically covers missing physics or an incorrect range.
Our experiment enumerates five equally weighted gains and minimizes their average exactly. It uses the expected-loss idea without random sampling, a neural network, or a training service. Moving the later test slider evaluates the fitted policies; it does not perform training.
The domain-randomization experiment changes a distracting input feature in a seeded training dataset. Compare its fitted weights and held-out errors to see when that intervention breaks a shortcut, and what it costs on familiar inputs.
Give the experiment a precise model
Let x be a point robot's position in meters, with a goal at zero. Its action u commands displacement in meters. The actuator gain g is dimensionless and remains constant during each rollout:
x[k+1] = x[k] + g u[k]
u[k] = −K x[k]
x[k+1] = (1 − gK)x[k]
K is a dimensionless policy coefficient. A gain g = 1 executes the requested displacement exactly. A gain g = 1.8 produces 1.8 times that displacement.
Every training and test episode starts at x[0] = 1 m. The model assumes exact position measurements and immediate action execution, with no noise, delay, saturation, inertia, or obstacles. It specifies no travel time between updates.
Each test runs for six updates. Success means |x[6]| ≤ 0.05 m. We also record the sum of squared commands, Σₖ₌₀⁵ u[k]², in m² as an effort proxy; this quantity is not physical energy.
Fit the two policy coefficients
Training minimizes squared position error after one update from 1 m. For gain g, that error is L(K,g) = (1 − gK)² m². Six-update test performance is a separate evaluation metric.
The nominal training distribution contains only g = 1. Its objective is (1 − K)², so the exact minimum is Knominal = 1, with zero training loss.
The varied training distribution assigns equal probability to g = 0.5, 0.75, 1, 1.25, and 1.5. Their mean is 1, and their mean square is 9/8. Expanding the average loss gives:
Lvaried(K) = 1 − 2K + (9/8)K²
dLvaried/dK = −2 + (9/4)K
Kvaried = 8/9 ≈ 0.888889
The positive quadratic coefficient makes this the unique global minimum. More generally, equal-weight gains give K = Σgᵢ / Σgᵢ²*. The varied policy's training loss is 1/9 ≈ 0.111111 m².
Those minimum training losses use different distributions, so they are not a common performance comparison. On the same varied training distribution, K = 1 scores 0.125 m² and K = 8/9 scores 0.111111 m². At the nominal gain alone, K = 1 reaches zero immediately, while K = 8/9 leaves x[1] = 1/9 m.
Test both policies on an unseen gain
Now hold the test gain at g = 1.8, outside the varied training range of 0.5 to 1.5. Neither training distribution contains this test setting. Keep each fitted K unchanged.
The nominal policy's first command is −1 m, giving x[1] = 1 − 1.8 = −0.8 m. The varied policy commands −8/9 m, giving x[1] = 1 − 1.8 × 8/9 = −0.6 m. Both overshoot and then alternate sides of the goal.
Their exact trajectories follow x[k] = (1 − gK)ᵏ m:
| Metric at g = 1.8 | Nominal training | Varied training |
|---|---|---|
| Discrete pole 1 − gK | −0.8 | −0.6 |
| Final distance after six updates (m) | 0.262144 | 0.046656 |
| Squared-command sum (m²) | 2.586890 | 1.231881 |
| Meets the 0.05 m goal tolerance | No | Yes |
The varied policy wins this particular test. We did not optimize the squared-command sum or the six-update distance during training. These are measured consequences of the chosen policy and test model.
For this scalar recurrence, error decays exactly when |1 − gK| < 1. With positive g and K, that means 0 < gK < 2. The nominal policy is stable for 0 < g < 2, and the varied policy for 0 < g < 2.25.
At either upper boundary, the pole is −1 and motion alternates without shrinking. At g = 2.5, both policies grow away from the goal: their final distances are 11.390625 m and 3.333505 m. The wider stability interval remains a property of this exact model, including its absence of delay and actuator limits.
Inspect the training and test curves
The first chart shows the two training objectives and their fitted minima. The second shows both policies acting on the same test model. Moving the test gain leaves the training curves and coefficients unchanged.
Start at g = 1.8, then choose Nominal model matches. The nominal policy reaches zero in one update. Try Both policies unstable to see why a broader training distribution still needs limits.
The replay selects a state after k updates. Final distances, outcomes, and command sums always cover all six updates. Chart axes expand to show growing responses, and the discrete pole determines each stability label.
Evaluate beyond the training distribution
Separate the stages of an evaluation:
- Training: select the policy using the stated model distribution and objective.
- Validation: choose ranges, model classes, or other design settings using separate trials.
- Final tests: hold those choices fixed and evaluate unseen episodes or regimes, with success criteria set beforehand.
- Hardware tests: report the actual robot, sensing, timing, limits, and measured outcomes, including failures.
The behavior-cloning covariate-shift lesson isolates a different mechanism: a learned policy changes the states it visits while the dynamics and expert rule stay fixed. Here, the actuator gain itself changes between simulated regimes.
A held-out regime can fall between training values or outside their range. The generalization condition map extends that distinction to two observed inputs, where familiar individual values can still form an unseen joint combination. With multiple parameters, hold out combinations as well as individual values. Use the same test conditions for competing policies so the comparison answers one question.
The slider supplies a deterministic simulation sensitivity study. Trying many gains and choosing a favorite result does not produce an unbiased hardware success rate. The lesson's exact formulas support the stated model conclusions; physical transfer would require separate measurements.
The distribution itself is a design choice. Wider variation can lower performance on a matched nominal system, and a mean objective can trade worse cases for better ones. The next exercise makes that tradeoff explicit.
Reproduce the comparison in Python
This standard-library script fits each policy from its training gains. Test gains enter only after fitting. It accumulates commands across the same six updates as the lab.
training = {
"nominal": [1.0],
"varied": [0.5, 0.75, 1.0, 1.25, 1.5],
}
def fit(gains):
return sum(gains) / sum(g * g for g in gains)
def test(k, gain):
x, effort = 1.0, 0.0
for _ in range(6):
u = -k * x
x += gain * u
effort += u * u
return abs(x), effort
policies = {name: fit(gains) for name, gains in training.items()}
for name, k in policies.items():
loss = sum((1 - g * k) ** 2 for g in training[name]) / len(training[name])
print(f"{name}: K={k:.6f}, own training loss={loss:.6f}")
for gain in [1.8, 1.0, 2.5]:
for name, k in policies.items():
distance, effort = test(k, gain)
print(f"g={gain:.1f} {name}: distance={distance:.6f}, "
f"command sum={effort:.6f}, goal={distance <= 0.05 + 1e-12}")
Expected output:
nominal: K=1.000000, own training loss=0.000000
varied: K=0.888889, own training loss=0.111111
g=1.8 nominal: distance=0.262144, command sum=2.586890, goal=False
g=1.8 varied: distance=0.046656, command sum=1.231881, goal=True
g=1.0 nominal: distance=0.000000, command sum=1.000000, goal=True
g=1.0 varied: distance=0.000002, command sum=0.800000, goal=True
g=2.5 nominal: distance=11.390625, command sum=102.997070, goal=False
g=2.5 varied: distance=3.333505, command sum=16.179604, goal=False
Try it yourself
Exercise 1. A test actuator moves 0.12 m after a command of 0.10 m. Assuming this constant-gain model, estimate g. From x[0] = 1 m, calculate the first position under each fitted policy and identify whether this gain appeared in training.
Show the gain and position calculation
The estimated gain is 0.12/0.10 = 1.2. The nominal policy gives x[1] = 1 − 1.2 = −0.2 m. The varied policy gives x[1] = 1 − 1.2 × 8/9 = −1/15 ≈ −0.066667 m. Gain 1.2 lies within the varied range but appears in neither training list. One measurement determines g only under the assumed noiseless, constant-gain model.
Exercise 2. Compare the worst one-update squared error across the five varied training gains for K = 1 and K = 8/9. Does minimizing the average also minimize the worst error here?
Show the average-versus-worst-case comparison
For K = 1, the largest squared error is 0.25 m² at gains 0.5 and 1.5. For K = 8/9, the largest error occurs at g = 0.5: (1 − 4/9)² = 25/81 ≈ 0.308642 m². The average-loss optimum has a larger worst error than the feasible K = 1 policy, so it cannot be the worst-case optimum.
The diffusion-policy lesson examines how a policy can generate multiple plausible actions from a conditional distribution. That richer action model still depends on its observations and training data; choosing a denoising sampler does not resolve the dynamics mismatch studied here.
Sources and further study
- Tobin et al., Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World (2017): randomized rendering, object localization on real images, and use in robotic grasping.
- Peng et al., Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (ICRA 2018): expected-return training across dynamics models, parameter variation, and physical puck-pushing experiments. The preprint first appeared in 2017.
Next, use model uncertainty to name one effect this toy plant excludes. Write the extra state, parameter, or timing rule you would need before claiming that your test covers it.