Domain randomization: break a training shortcut

Change what training lets the model rely on

Domain randomization varies the training environment so a model must handle more than one set of conditions. A policy can learn a shortcut when one visual cue or simulator detail predicts the right action during training. That shortcut can fail when the relationship changes.

This lesson performs a controlled training intervention. Two policies receive the same positions, expert labels, example count, and fitting method. Only one training set randomizes a nuisance feature, then both policies face the same held-out tests.

The experiment uses abstract numeric features. It isolates a possible shortcut mechanism without claiming to render images, train a vision network, or transfer a policy to hardware.

Separate visual and dynamics randomization

Tobin and colleagues' 2017 domain-randomization paper varied rendered scene properties, including textures, lighting, camera properties, and distractors. The authors trained object-localization networks on synthetic images, tested real images, and used their detector in robotic grasping. Their ablations examined which training variations mattered for that setup.

Visual randomization changes the observations a model receives. A texture can change while an object's target location stays fixed. Other changes, such as moving the object or camera, can also change the correct label, depending on the output coordinate convention.

Peng and colleagues' 2018 dynamics-randomization paper varied properties including mass, damping, friction, controller gains, and action timing for a robotic-arm pushing task. The authors trained in simulation and evaluated on a physical Fetch arm. Those experiments change how actions affect motion, as well as selected observation properties.

The sim-to-real transfer lesson explores that dynamics question with changed actuator gains. Here, motion mechanics stay fixed while a nuisance feature changes during supervised policy fitting.

Give the shortcut a precise model

A point robot starts at x meters and must reach zero in one action. The environment applies the commanded displacement exactly:

Define a dimensionless position feature z = x / ℓ, with fixed scale ℓ = 1 m. The policy also observes a dimensionless nuisance feature c. Its action rule is:

Both weights are dimensionless. The normalized expert action is y = u⋆/ℓ = −z. The nuisance does not affect the actual position, dynamics, or expert rule, so changing c leaves the label valid.

Both policies receive exact position directly. There is no perception system, noise on z, motion duration, saturation, contact, or obstacle. Supplying such a useful feature makes this a controlled model of reliance on a shortcut, not a model of extracting position from pixels.

Hold the training comparison fixed

Choose N training positions at evenly spaced bin midpoints:

The lab offers N = 16, 32, or 64. Both training sets use every position once and pair it with yᵢ = −zᵢ. Increasing N changes the midpoint grid for both fits; it does not retain the smaller grid as a subset.

Nominal training sets cᵢ = zᵢ. The nuisance perfectly predicts the expert label there, even though it has no role in the task mechanics.

Randomized training draws rᵢ uniformly from −1 to 1 in the ideal sampling model and uses:

At α = 0, the two datasets match exactly. At α = 1, the nominal relationship disappears from the data-generation rule. Intermediate strengths retain some dependence on z.

A seeded integer generator makes the finite draws reproducible. Both policies still receive N labels and use the same model and penalty. This comparison changes the training inputs without adding demonstrations to one side.

Fit the same ridge objective twice

We fit a linear policy without an intercept. Both weights minimize mean normalized squared action error plus a ridge penalty:

Here λ is dimensionless and strictly positive. The lab offers 0.001, 0.01, and 0.1. The penalty makes the solution unique even when the nominal feature columns are identical.

With X containing rows [zᵢ, cᵢ], the fitted weights solve:

The implementation solves this two-by-two system. It performs a new fit when a training control changes. Selecting a test relation or inspecting a test position leaves both fits frozen.

Calculate the nominal shortcut

In nominal training, c = z. Only the sum w_z + w_c determines the predictions. For a fixed sum, equal weights minimize their squared penalty, so ridge splits the reliance equally.

Let V = mean(z²). The exact solution is:

For N = 32, V = (N² − 1)/(3N²) = 0.3330078125. At λ = 0.01, both weights equal −0.4926037213. The nominal training action RMSE is about 0.008536 m.

Now reverse the nuisance relationship to c = −z. The two weighted contributions cancel, giving u = 0. At x = 0.75 m, the robot stays at 0.75 m, despite the small nominal training error.

With full randomization, seed 7, and the same N and λ, the fitted weights are w_z = −0.9701741618 and w_c = 0.0042638021. At that same reversed test condition, the command is −0.730828473 m and the endpoint is 0.019171527 m. The finite fit retains a small nuisance weight; randomization does not force it to exactly zero.

Freeze familiar and shifted evaluation sets

Both policies face 20 fixed positions zⱼ = −1 + (2j + 1)/20, for j from 0 through 19. None of these positions occurs in any offered training grid. Each test relation uses those same positions and expert labels:

• Familiar relation: c = z. Positions are held out, while the nominal feature relationship stays familiar.
• Reversed relation: c = −z. The nuisance relationship changes while the correct expert action stays y = −z.

We score one-step endpoint RMSE, √mean((x after − 0)²), in meters. For this exact reaching model, endpoint error equals action-prediction error in meters. The two metrics coincide because the actuator applies u exactly.

The default results are:

Fixed test set	Nominal fit RMSE (m)	Randomized fit RMSE (m)
Familiar relation	0.008530	0.019657
Reversed relation	0.576628	0.014740

The randomized fit handles this reversal better and has a larger familiar-relation error. Report both. Its own training RMSE, 0.017005 m, also scores different inputs from the nominal training RMSE; those training numbers do not provide a matched test comparison.

These deterministic test grids have no stated deployment frequencies. They supply a controlled error comparison, without a population success rate or confidence interval. The generalization lesson explains why a coverage claim needs named conditions.

Inspect the domain randomization experiment

The first plot shows the actual feature pairs that enter fitting. Blue circles lie on c = z. Amber squares show the changed nuisance feature at the same training positions.

Try the following comparisons:

• Select No randomization. The data, weights, and outcomes must agree. This is the ablation that removes the intervention.
• Select Weak randomization. At α = 0.25, the default randomized fit still uses nuisance weight −0.184416 and has reversed RMSE 0.196390 m.
• Select Stronger ridge penalty. Both fits receive λ = 0.1. The randomized reversed RMSE rises to 0.119514 m in this example.
• Change Training seed. Finite randomized samples change; nominal training and both test sets remain fixed.
• Switch Test nuisance relation. The plots and inspected action change without refitting either policy.

Change training inputs, then freeze both fits

Can randomization break a learned shortcut?

Both policies get the same positions, expert labels, example count, model, and ridge penalty. Nominal training sets nuisance c = z. Randomized training uses c = (1 − α)z + αr, with a seeded draw r from −1 to 1.

Randomization presetCustom trainingFull randomizationNo randomizationWeak randomizationStronger ridge penalty

Training examples per policy16 examples32 examples64 examples

Randomization strength

1.000000

Training seed

7

Ridge penalty0.0010.0100.100

Training controls refit both policies. The test sets always contain the same 20 held-out positions. Inspecting a test relation or position changes no training data or fitted weight. Each normalized feature and weight is dimensionless; one action unit means 1 m of displacement.

Reset domain randomization lab

Training examples per fit

32

Nominal position weight

-0.492604

Nominal nuisance weight

-0.492604

Randomized position weight

-0.970174

Randomized nuisance weight

0.004264

Nominal training action RMSE (m)

0.008536

Randomized training action RMSE (m)

0.017005

Nominal familiar endpoint RMSE (m)

0.008530

Randomized familiar endpoint RMSE (m)

0.019657

Nominal reversed endpoint RMSE (m)

0.576628

Randomized reversed endpoint RMSE (m)

0.014740

Each training RMSE scores its own input set. The familiar and reversed test RMSEs compare both fits on identical inputs within that test. They are errors for a fixed test grid, without a success threshold or statistical interval.

Test nuisance relationFamiliar: c = zReversed: c = -z

Inspect held-out position

0.750000 m

The training intervention in feature space

Held-out position (m)

0.750000

Test nuisance value

-0.750000

Expert displacement (m)

-0.750000

Nominal displacement (m)

0.000000

Randomized displacement (m)

-0.730828

Nominal endpoint error (m)

0.750000

Randomized endpoint error (m)

0.019172

Endpoint error on the fixed test set

Only the training nuisance changes between fits. Finite samples and ridge regularization can leave a nonzero nuisance weight; held-out behavior must still be checked.

The second plot joins the 20 test conditions. It shows endpoint error across starting positions, not a time trajectory. Its vertical scale changes with the selected relation and fits, so compare the labeled values as well as the curve shapes.

State what randomization can leave unresolved

A useful randomization scheme must preserve the intended task and valid supervision. This example can keep its labels because c does not change the correct action. Changing relevant geometry, coordinate frames, or mechanics can require new labels or a different reward calculation.

The distribution also matters. Randomizing one cue leaves other shortcuts available. A training range can miss deployment conditions, and unrealistic combinations can spend a limited training budget on cases that do not support the intended task.

This lab supplies an exact useful feature, uses a linear model family, and fits noiseless expert labels. Those choices make the mechanism tractable. They leave perception errors, missing physics, and long rollout effects untested.

Domain randomization varies training conditions. A separate adaptation step can use target-domain data to update a model or representation; both approaches can appear in one workflow. A recurrent policy can also adjust its behavior through internal state while keeping its learned weights fixed, as discussed by Peng and colleagues.

Record which mechanism your system uses. If you choose the randomization strength by inspecting these held-out scores, treat that set as validation and reserve another test for the final claim. Robot data leakage and policy evaluation cover those evidence boundaries.

Reproduce the training intervention in Python

This standard-library program generates the default paired training sets, fits both ridge policies, and evaluates the fixed test grids. It reports nominal results first and randomized results second. The training seed never generates or changes test positions.

from math import sqrt

n, seed, alpha, penalty = 32, 7, 1.0, 0.01
nominal, randomized = [], []
for i in range(n):
    z = -1 + (2*i + 1)/n
    seed = (1664525*seed + 1013904223) % (2**32)
    r = 2*seed/(2**32) - 1
    nominal.append((z, z, -z))
    randomized.append((z, (1-alpha)*z + alpha*r, -z))

def fit(rows):
    count = len(rows)
    a = sum(z*z for z, c, y in rows)/count + penalty
    b = sum(z*c for z, c, y in rows)/count
    d = sum(c*c for z, c, y in rows)/count + penalty
    p = sum(z*y for z, c, y in rows)/count
    q = sum(c*y for z, c, y in rows)/count
    determinant = a*d - b*b
    return ((p*d-q*b)/determinant, (q*a-p*b)/determinant)

def rmse(weights, rows):
    wz, wc = weights
    return sqrt(sum((wz*z + wc*c - y)**2 for z, c, y in rows)/len(rows))

wn, wr = fit(nominal), fit(randomized)
print(f"nominal weights: {wn[0]:.6f}, {wn[1]:.6f}")
print(f"randomized weights: {wr[0]:.6f}, {wr[1]:.6f}")
print(f"own training RMSE: {rmse(wn, nominal):.6f}, {rmse(wr, randomized):.6f} m")
for name, sign in [("familiar", 1), ("reversed", -1)]:
    test = [(-1+(2*i+1)/20, sign*(-1+(2*i+1)/20),
             1-(2*i+1)/20) for i in range(20)]
    print(f"{name} endpoint RMSE: {rmse(wn, test):.6f}, {rmse(wr, test):.6f} m")
z, c = 0.75, -0.75
for name, weights in [("nominal", wn), ("randomized", wr)]:
    u = weights[0]*z + weights[1]*c
    print(f"{name} reversed x=0.75: u={u:.6f} m, endpoint={z+u:.6f} m")

Expected output:

nominal weights: -0.492604, -0.492604
randomized weights: -0.970174, 0.004264
own training RMSE: 0.008536, 0.017005 m
familiar endpoint RMSE: 0.008530, 0.019657 m
reversed endpoint RMSE: 0.576628, 0.014740 m
nominal reversed x=0.75: u=0.000000 m, endpoint=0.750000 m
randomized reversed x=0.75: u=-0.730828 m, endpoint=0.019172 m

Check two randomization decisions

Exercise 1. A policy has dimensionless weights w_z = w_c = −0.45. At x = 0.8 m, calculate its displacement and endpoint under c = z and under c = −z. What does this show about evaluating only the familiar relationship?

Check the shortcut calculation

Here z = 0.8. With c = 0.8, u = 1 m × (−0.45 × 0.8 − 0.45 × 0.8) = −0.72 m, so the endpoint is 0.08 m.

With c = −0.8, the weighted terms cancel. The action is 0 m and the endpoint stays at 0.8 m. Testing the familiar relationship alone leaves this changed relationship unevaluated.

Exercise 2. For one training row, z = 0.6, r = −0.2, and α = 0.25. Compute c and the normalized expert label. Does the intervention require another expert label for this row? What changes if you select α by repeatedly checking a held-out score?

Check the randomized row and evaluation boundary

The nuisance becomes c = 0.75 × 0.6 + 0.25 × (−0.2) = 0.4. The label remains y = −0.6, because the nuisance does not affect the task rule. The fit still uses one labeled example for this position.

Choosing α from the held-out score makes that set a validation set for the choice. Keep a separate final test if you want evidence beyond that selection process.

Sources and a test to design next

• Tobin and colleagues, Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World, 2017, with the original paper's visual-randomization method and ablations.
• Peng and colleagues, Sim-to-Real Transfer of Robotic Control with Dynamics Randomization, ICRA 2018, including randomized dynamics, recurrent policies, and physical pushing experiments.

Choose one nuisance your own evaluation should vary, and state which labels must stay valid. Then use the policy-evaluation workflow to test complete outcomes under declared conditions.