Diffusion Policy: generate robot actions by denoising

Diffusion Policy generates robot actions by repeatedly refining a noisy candidate. Observations condition that refinement, so the policy can represent several possible responses to one situation. This lesson makes the denoising calculation visible with a small, fully specified distribution.

The lab uses one normalized action value and an exact mathematical denoiser. It demonstrates sampling mechanics without training a neural network or simulating a robot.

Represent actions conditioned on observations

Cheng Chi and colleagues introduced Diffusion Policy at RSS 2023. Their formulation models an action sequence conditioned on recent observations. A learned denoiser refines noise into that sequence, and the robot executes a prefix before replanning from new observations.

Write the conditional distribution as p(A | O). Here O holds the observation context and A holds the proposed actions. The authors' project page shows their architecture and task demonstrations.

Diffusion Policy fits within behavior cloning: demonstration data supply the training examples. Behavior cloning can use many output distributions. A deterministic predictor trained with squared action error is one baseline, not the definition of all imitation learning.

Understand why an average can miss both choices

Suppose demonstrations contain two action clusters: one near −1, another near +1. These normalized values could represent two choices of action offset. Their physical meaning would depend on the robot's action interface.

With equal weights, the conditional mean is 0. A squared-error point predictor minimizes its expected loss at that mean, even when demonstrations rarely choose values near zero. A multimodal distribution keeps the separate clusters visible.

In our example, each cluster has standard deviation 0.12. The density at zero is about 2.77 × 10⁻¹⁵, while the density near either mode is about 1.66. Averaging the choices lands in an extremely low-density part of this distribution.

This example says nothing about obstacles or task success. A scalar action value has no geometry, contact model, or collision checker. Multiple peaks describe the demonstrated choices; they do not certify those choices as safe.

Train on deliberately corrupted demonstrations

Denoising training begins with a clean data example and a known random corruption. The standard Gaussian construction from Ho, Jain, and Abbeel's DDPM paper is:

Here a₀ is a clean action, x is its noisy version, and α controls the retained signal. This lesson writes α for the cumulative signal fraction, often written ᾱ in diffusion notation. A small α leaves mostly noise.

For a training example, the noise ε is known because the training procedure generated it. The network receives the noisy action, noise level, and observation context. A common objective penalizes squared error between the added noise and its prediction:

Gradient descent updates the network parameters θ during training. Under squared loss, the ideal predictor estimates conditional mean noise. It cannot generally identify the particular noise draw uniquely from one corrupted example.

Generate a candidate through denoising

At inference, begin with random noise and hold the current observation context fixed during a sampling call. Repeated denoising updates produce a candidate action sequence. The network parameters stay fixed during these updates; sampling iterations do not retrain the model.

Several samplers can use a noise predictor. This lab uses the deterministic DDIM update from Jiaming Song, Chenlin Meng, and Stefano Ermon, with the stochastic sampling parameter η set to zero. New initial noise can still produce a different result.

The Diffusion Policy paper, Section III-D, also discusses DDIM for faster inference. Its reported hardware timings belong to its experiments; this browser example does not reproduce those measurements.

Define a small action distribution exactly

The lab's context specifies w, the probability of selecting the right mixture component. The left component has probability 1 − w:

This is an original teaching distribution with a known formula. The slider supplies w directly; no camera or language model infers it. Its global mean is 2w − 1.

After Gaussian corruption, the two component means become −√α and +√α. Both components have variance V = ατ² + 1 − α. The dashed density in the lab shows that exact forward marginal.

A learned policy must infer a useful distribution from data. Here, knowing the distribution lets us calculate its ideal denoiser and check each step independently. The uncertainty lesson explains why a variance and a standard deviation represent different quantities.

Calculate an analytic denoising update

Given noisy value x, Bayes' rule gives the right-mode responsibility r. It is the probability of the right mixture component under this specified model:

The sigmoid is 1 / (1 + exp(−u)) for input u. Conditioning each Gaussian component gives the clean-action mean:

For w = 0.5, α = 0.25, and x = 0.3, V = 0.7536. Then r ≈ 0.598228 and m ≈ 0.198385. This intermediate estimate can lie between modes because the noisy value still leaves ambiguity.

Recover the corresponding predicted noise, then advance to a larger signal fraction β:

With β = 0.36, ε̂ ≈ 0.231873, giving xnext ≈ 0.304529. This is a deterministic DDIM step. At the final β = 1 step, xnext equals the previous clean-action estimate; no division by zero is needed.

Replay one sample and change its context

The experiment uses 12 updates with these signal fractions, listed in sampling order:

0.001, 0.01, 0.04, 0.09, 0.16, 0.25, 0.36,
0.49, 0.64, 0.81, 0.90, 0.97, 1.00

It starts from seeded standard-normal noise. At α = 0.001, the true forward marginal is close to that normal distribution, but still differs from it. The starting approximation and finite update schedule mean that this sampler does not reproduce the target distribution exactly, even with the exact denoiser.

Denoising experiment

Turn seeded noise into one action sample

This teaching distribution has two modes, centered at -1 and +1 with standard deviation 0.12. Its denoiser is known analytically; no neural network or robot runs here.

Right-mode probability

0.5

Denoising update

4 of 12

Next denoising updateFinish denoisingNew noise sampleReset diffusion experiment

The context slider changes the target mixture weights while keeping this noise seed. New noise advances the reproducible seed and restarts at update 0. Denoising updates are computations inside one policy call.

Target distribution and current noise level

Refine one candidate action

Noise seed

5

Denoising update

4 of 12

Signal fraction

0.160000

Noise scale

0.916515

Current noisy value

-0.187782

Estimated clean action

-0.089981

Right-mode responsibility

0.455530

Target mean

0.000000

Sampling status

Denoising

Responsibility means the posterior probability of the right mixture component given this noisy value. The 12 deterministic DDIM updates and standard-normal starting approximation do not guarantee exact target-distribution samples. At update 12, the noise scale is zero and “Current noisy value” holds the final sample. Readouts round to six decimal places.

With w = 0.5 and seed 5, the starting value is −0.190315. After four updates, it is −0.187782, with estimated clean action −0.089981. Finish the sequence to obtain −0.906935.

Keep seed 5 and raise w to 0.9. The final value becomes +0.972843. Reset, then choose New noise sample: seed 6 produces +0.998159 after finishing at w = 0.5.

These examples show dependence on context and initialization. A handful of outputs cannot establish correct mixture frequencies. The plotted densities come from the exact formulas, and are not histograms of generated samples.

Separate denoising steps from robot time

A real action chunk contains several future commands. Diffusion Policy jointly generates a chunk, executes only a chosen prefix, then refreshes its observations and replans. This is receding-horizon execution, described in the original paper's action-sequence formulation.

For a timing example, suppose the robot applies commands at 10 Hz, predicts four commands, and executes two before replanning. Assume zero inference latency for this table. The letters distinguish newly generated chunks.

Observation time	Predicted chunk	Commands executed	Unused tail
0.0 s	a₀, a₁, a₂, a₃	a₀ during 0.0–0.1 s; a₁ during 0.1–0.2 s	a₂, a₃
0.2 s	b₂, b₃, b₄, b₅	b₂ during 0.2–0.3 s; b₃ during 0.3–0.4 s	b₄, b₅
0.4 s	c₄, c₅, c₆, c₇	c₄ during 0.4–0.5 s; c₅ during 0.5–0.6 s	c₆, c₇

Twelve denoising updates would occur inside each sampling call. They do not mean twelve robot commands or twelve seconds. Real scheduling must account for observation age, computation latency, and the low-level controller.

Action chunking also appears in Zhao and colleagues' Action Chunking with Transformers, or ACT. ACT uses a conditional variational autoencoder and transformer architecture. Chunking and diffusion are separate design choices. The imitation-learning methods overview also distinguishes the action-model architecture from the training objective and access to expert labels.

Check the limits before interpreting a sample

The lab assumes the clean distribution and exact denoiser are already known. It omits learned perception, training data, full action sequences, robot dynamics, and physical execution. Its left and right labels refer only to mixture components.

A deployed policy needs checks beyond a plausible-looking action:

• Data coverage: test new starting states, objects, and disturbances beyond demonstration conditions.
• Action interface: verify normalization, units, frame conventions, and actuator limits.
• Timing: measure inference latency and the age of the observations that condition each chunk.
• Execution: evaluate task success and failure recovery across full episodes.

The robot-data leakage lesson shows why nearby frames from the same recording cannot stand in for unseen episodes. DAgger addresses a separate collection question: which expert labels should be added after observing where a learner goes? These data decisions apply beyond any particular action generator.

More denoising steps alone cannot supply missing demonstrations or correct a camera calibration error. Joint sequence prediction does not guarantee smooth, feasible, or collision-free motion. The sim-to-real lesson examines these deployment gaps.

A vision-language-action model describes how visual and language information relate to actions. A diffusion action generator can be one component within a larger model. Adding denoising does not by itself provide language grounding or a general-purpose robot policy.

Reproduce the sampler in Python

This standard-library script reproduces the mixture denoiser, seeded noise, and 12 deterministic updates. The seed produces repeatable pseudo-random values; it does not select a hardcoded action or mixture component.

from math import cos, exp, log, pi, sqrt

schedule = [.001, .01, .04, .09, .16, .25, .36,
            .49, .64, .81, .90, .97, 1.0]


def noise(seed):
    state = (1664525 * seed + 1013904223) % 2**32
    u = (state + .5) / 2**32
    state = (1664525 * state + 1013904223) % 2**32
    v = (state + .5) / 2**32
    return sqrt(-2 * log(u)) * cos(2 * pi * v)


def denoise(x, alpha, weight):
    variance = alpha * .12**2 + 1 - alpha
    odds = log(weight / (1 - weight)) + 2*sqrt(alpha)*x/variance
    right = 1 / (1 + exp(-odds))
    gain = .12**2 * sqrt(alpha) / variance
    mean = gain*x + (1 - gain*sqrt(alpha))*(2*right - 1)
    return mean, right


def sample(weight, seed):
    x = noise(seed)
    for step, alpha in enumerate(schedule[:-1]):
        mean, right = denoise(x, alpha, weight)
        predicted_noise = (x - sqrt(alpha)*mean) / sqrt(1 - alpha)
        next_alpha = schedule[step + 1]
        x = sqrt(next_alpha)*mean + sqrt(1 - next_alpha)*predicted_noise
    return x


mean, right = denoise(.3, .25, .5)
predicted_noise = (.3 - sqrt(.25)*mean) / sqrt(.75)
next_value = sqrt(.36)*mean + sqrt(.64)*predicted_noise
print(f"worked responsibility={right:.6f}, clean={mean:.6f}")
print(f"worked noise={predicted_noise:.6f}, next={next_value:.6f}")
print(f"seed 5 starts={noise(5):.6f}")
for weight, seed in [(.5, 5), (.9, 5), (.5, 6)]:
    print(f"w={weight:.1f}, seed={seed}: final={sample(weight, seed):.6f}")

Expected output:

worked responsibility=0.598228, clean=0.198385
worked noise=0.231873, next=0.304529
seed 5 starts=-0.190315
w=0.5, seed=5: final=-0.906935
w=0.9, seed=5: final=0.972843
w=0.5, seed=6: final=0.998159

Check the distribution and timing

Exercise 1: Separate a mixture mean from an action sample

Let the right component have weight 0.8, the left weight 0.2, and both have standard deviation 0.12. Calculate the distribution's mean and variance. Does the mean identify a typical value within either component?

Solution: The mean is 0.2(−1) + 0.8(+1) = 0.6. The variance is the within-component variance plus the variance of component means: 0.12² + 4(0.8)(0.2) = 0.6544. The mean lies 0.4/0.12 ≈ 3.33 standard deviations below the right component's center, so it does not summarize a typical within-component action.

Exercise 2: Count denoising calculations and executed commands

Use the timing table's 10 Hz command rate and two executed commands per chunk. Each new chunk requires 12 denoising-network evaluations. Ignoring latency, how many sampling calls, denoising evaluations, and executed commands occur during one second of steady operation?

Solution: Replan every 0.2 s, giving 5 sampling calls per second. Each call requires 12 evaluations, so the policy performs 60 denoising evaluations and executes 10 commands during that second. Four-command prediction produces 20 candidate commands across those calls; the execution schedule uses ten and discards the remaining tails.

Sources and further study

• Cheng Chi and colleagues, Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023 version: conditional action diffusion, visual conditioning, action horizons, and DDIM inference. The project page includes the authors' demonstrations and code links.
• Jonathan Ho, Ajay Jain, and Pieter Abbeel, Denoising Diffusion Probabilistic Models: Gaussian corruption and the noise-prediction training objective.
• Jiaming Song, Chenlin Meng, and Stefano Ermon, Denoising Diffusion Implicit Models: deterministic sampling and the distinction between training and inference schedules.
• Tony Z. Zhao and colleagues, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware: ACT's action chunks and conditional variational-autoencoder architecture.

Use the behavior-cloning evaluation example next to separate a model's prediction error from what happens when its own actions change future observations.