explainer
Vision-language-action models: from observations to robot commands
Learn how vision-language-action models connect images and instructions to robot actions. Compare token and continuous outputs, then inspect how units, frames, and timing change a synthetic command.
What you will learn
- Describe how a VLA uses observations and language to predict robot actions.
- Compare discrete action tokens with continuous action generation.
- Distinguish policy training from inference and physical execution.
- Decode a normalized command with stated units, frame, and timing.
- Evaluate robot-policy claims using the tested setup and adaptation conditions.
Before you start
A vision-language-action model, or VLA, predicts robot actions from visual observations and a language instruction. The output still needs a physical meaning: which robot quantity it commands, in which units and coordinate frame, and for how long.
This lesson compares two published designs from 2024, OpenVLA and π₀. Then you will decode a small synthetic action packet and see how changing its interface changes the motion. The inspector runs no trained model and makes no claim about task success.
What a vision-language-action model does
Imagine a camera view containing a cup and a bowl, with the instruction “place the cup in the bowl.” A VLA conditions its action prediction on those inputs. It may predict one action or a sequence of future actions, depending on its design.
A compact description is a ∼ πθ(· | o, ℓ): a policy with learned parameters θ predicts an action a given observation o and language instruction ℓ. The notation allows a distribution over actions. An action could specify an end-effector displacement, a joint target, or another robot-specific command.
The OpenVLA project gives a concrete example of adapting a pretrained vision-language model to robot action prediction. A VLA describes a class of learned policies. It does not specify one universal camera interface, action format, or controller. The robot foundation-model comparison examines a different question: which data and robot embodiments support a pretrained policy, what adaptation it needs, and which transfer claims its evaluation supports.
Name the observations and instruction
Before comparing models, identify what they actually receive:
| Input | Example | Interface question |
|---|---|---|
| Visual observation | A front camera image | Which camera, crop, resolution, and capture time? |
| Language | “Place the cup in the bowl” | Which object and task does the instruction identify? |
| Robot state, if supported | Joint positions or gripper state | Which order, units, and timestamp? |
| History, if supported | Earlier images or states | How much past context reaches the model? |
These rows are possible inputs, not a checklist every VLA satisfies. Original OpenVLA uses a single image and language instruction. The original π₀ paper includes multiple images, language, and robot joint-state inputs.
Language helps specify the intended behavior. It does not supply a measured object pose or guarantee that the policy grounds the instruction correctly. The NLP lesson introduces language representations; robotic action also depends on the observed physical situation.
Compare action tokens and continuous outputs
The 2024 OpenVLA design combines SigLIP and DINOv2 visual features, a projection into a language-model embedding space, and a Llama 2 backbone. Its output uses discrete action tokens. The authors' architecture description explains those components.
The original OpenVLA paper discretizes each action dimension into 256 bins. It predicts token IDs, then decodes them into numerical actions. Those IDs are an encoding of a control value; they are not natural-language instructions for a motor.
The original π₀ model adds an action expert to a PaliGemma-based backbone. It uses conditional flow matching to generate continuous action chunks. During generation, a learned vector field transforms a noise sample into an action sequence.
A transformer can use continuous action embeddings as sequence elements, so the word “token” alone does not establish that a model quantizes its output. Read the action decoder and training objective. The diffusion-policy lesson examines another continuous generative action policy.
Learn an action rule from robot data
Behavior cloning learns actions from demonstrations. A VLA training record can connect an observation and instruction to the demonstrated action. Training adjusts model parameters to improve predictions across those records.
OpenVLA trains action-token predictions with cross-entropy loss on the action tokens. π₀ trains its action-generation component with a flow-matching objective. These objectives match their chosen output representations. Both examples start from pretrained vision-language components and use robot data to learn action behavior. (OpenVLA, π₀)
Task-specific adaptation uses additional training data and parameter updates. Keeping those adaptation demonstrations separate from evaluation episodes helps distinguish learning the target setup from handling new conditions. The robot-data splitting experiment shows how shared recordings can make a score answer an easier question. A low prediction loss on familiar demonstrations does not measure success during an independent robot rollout. Specify which conditions are unseen, then use a policy-evaluation protocol that reports full episode outcomes.
Separate inference from physical execution
At inference, a deployed policy uses its learned parameters to predict actions from new inputs. A controller then interprets and executes the output. New observations can inform later predictions, closing the loop described in feedback control.
Action chunking predicts several future commands together. A system may execute some or all of that chunk before requesting another prediction. A chunk's command rate, the rate of new policy predictions, and camera capture rate are distinct quantities.
π₀'s original implementation generates action chunks through multiple flow-matching integration steps. Those numerical generation steps are not robot motion intervals. The paper distinguishes the sampling computation from executing the resulting chunk. (π₀, inference appendix)
A useful timing record includes observation capture time, prediction completion time, and scheduled command times. Their differences expose stale observations and execution delays. This lesson's inspector models only a command schedule, with no inference latency or sensor stream.
Give the action an explicit interface
A normalized pair such as z = (0.5, −0.25) has no physical units by itself. Our original teaching interface defines two possible meanings:
| Interpretation | Decode each component | Execution assumption |
|---|---|---|
| Velocity | v = 0.2z m/s | Hold the velocity for one period Δt |
| Position increment | δ = 0.05z m | Complete that displacement over Δt at constant velocity |
The scales bound each component independently. A packet with both normalized components equal to 1 has vector magnitude √2 times the component scale. These example scales are not OpenVLA or π₀ normalization statistics.
The frame also belongs in the contract. For a robot-frame vector and fixed heading θ, rotate it into world coordinates:
w = R(θ)b
R(θ) = [[cos θ, −sin θ], [sin θ, cos θ]]
Velocities and displacement vectors need this rotation without an added origin translation. An absolute position target would need the full point transformation, including translation. Review homogeneous transformations for that distinction.
Decode and rotate one action packet
Use z = (0.5, −0.25), robot heading θ = 90°, and Δt = 0.2 s. Under the velocity interpretation, the robot-frame velocity is (0.1, −0.05) m/s. At 90°, R maps (x, y) to (−y, x):
v_world = (0.05, 0.10) m/s
δ_world = v_world Δt = (0.01, 0.02) m
Five identical packets take 1 s and move the ideal point from the origin to (0.05, 0.10) m. Each displacement has length √(0.01² + 0.02²) = 0.022361 m.
Under the position-increment interpretation, the decoded robot-frame increment is (0.025, −0.0125) m. Rotation gives (0.0125, 0.025) m per packet, reaching (0.0625, 0.125) m after five packets. Completing each increment in 0.2 s implies world velocity (0.0625, 0.125) m/s under our ideal interpolation rule.
Inspect the command in the lab
Start with Robot-frame velocity, then increase the action period from 0.2 to 0.4 s. The velocity stays fixed, the five-packet duration doubles to 2 s, and the endpoint becomes (0.1, 0.2) m.
Choose Position increments and make the same period change. The endpoint stays at (0.0625, 0.125) m, while the implied velocity halves to (0.03125, 0.0625) m/s. A real position controller could follow a different interpolation and encounter limits; the diagram assumes exact constant-rate motion.
Choose World-frame velocity to use the components directly in the world frame. The default packet then reaches (0.1, −0.05) m. Changing the robot heading rotates the gray orientation markers but leaves that world-frame command unchanged.
Every example repeats one synthetic packet five times with fixed orientation. It omits images, language inference, learned behavior, contact, robot dynamics, and controller limits. It tests the meaning of a command, so its straight path supplies no evidence of VLA task competence.
Read evaluation results with their conditions
The 2024 OpenVLA paper separates direct robot evaluations from adaptation to new setups. Its Franka adaptation experiments use 5 Hz and 15 Hz controllers, depending on the setup. Those conditions belong with any claim about the result. (Original evaluation description)
When reading a robot-policy result, check:
- Robot and interface: embodiment, cameras, state inputs, action units, controller, and timing.
- Training access: whether the policy saw that robot, task, scene, or object during training or adaptation.
- Evaluation split: what changed, how many trials ran, and how the authors defined success.
- Execution: whether a person intervened, which part of an action chunk ran, and what compute supported it.
- Failure modes: misses, dropped objects, wrong targets, and recovery behavior across trials.
A result on one task suite does not establish a general success rate for every robot. The examples here describe dated designs and interfaces, without ranking current models. The broader physical-AI lesson places learned policies alongside sensing, control, and physical execution.
Reproduce the packet calculation in Python
This standard-library example implements the same synthetic contract. It uses the exact 90° rotation identities so the unit and timing calculations remain easy to inspect.
from math import hypot
def inspect(mode, period, frame="robot"):
zx, zy = 0.5, -0.25
scale = 0.2 if mode == "velocity" else 0.05
x, y = scale*zx, scale*zy
# Robot heading is fixed at 90 degrees.
wx, wy = (-y, x) if frame == "robot" else (x, y)
dx, dy = (wx*period, wy*period) if mode == "velocity" else (wx, wy)
vx, vy = dx/period, dy/period
return dx, dy, vx, vy, 5*dx, 5*dy, 5*period
def report(label, mode, period, frame="robot"):
dx, dy, vx, vy, x, y, duration = inspect(mode, period, frame)
print(f"{label}: end=({x:.6f}, {y:.6f}) m, time={duration:.6f} s")
return dx, dy, vx, vy
dx, dy, vx, vy = report("Velocity", "velocity", 0.2)
print(f"World velocity=({vx:.6f}, {vy:.6f}) m/s")
print(f"Distance per packet={hypot(dx, dy):.6f} m")
report("Velocity, longer period", "velocity", 0.4)
report("Increment", "delta", 0.2)
_, _, vx, vy = report("Increment, longer period", "delta", 0.4)
print(f"Slower increment velocity=({vx:.6f}, {vy:.6f}) m/s")
report("World-frame velocity", "velocity", 0.2, "world")
Expected output:
Velocity: end=(0.050000, 0.100000) m, time=1.000000 s
World velocity=(0.050000, 0.100000) m/s
Distance per packet=0.022361 m
Velocity, longer period: end=(0.100000, 0.200000) m, time=2.000000 s
Increment: end=(0.062500, 0.125000) m, time=1.000000 s
Increment, longer period: end=(0.062500, 0.125000) m, time=2.000000 s
Slower increment velocity=(0.031250, 0.062500) m/s
World-frame velocity: end=(0.100000, -0.050000) m, time=1.000000 s
Try two action-interface checks
Exercise 1. Select velocity, robot frame, z = (1, 0), heading 180°, and period 0.1 s. What are the world velocity, displacement per packet, final position, and total scheduled duration?
Check the rotated velocity command
The decoded robot-frame velocity is (0.2, 0) m/s. A 180° rotation gives world velocity (−0.2, 0) m/s. Each period moves (−0.02, 0) m, so five packets finish at (−0.1, 0) m after 0.5 s. Tiny nonzero y values from floating-point sine calculations should round to zero at the displayed precision.
Exercise 2. Select position increments in the world frame with z = (1, −1). Compare periods 0.1 and 0.5 s. Does changing heading affect the endpoint? Is the per-packet distance 0.05 m?
Check timing and component bounds
Each packet commands (0.05, −0.05) m, independent of heading. Five packets reach (0.25, −0.25) m with either period. Duration changes from 0.5 to 2.5 s, and the implied velocity changes from (0.5, −0.5) to (0.1, −0.1) m/s. Each packet travels √(0.05² + 0.05²) = 0.070711 m because the 0.05 m scale bounds each component separately.
Sources and further study
- Kim and colleagues, OpenVLA project, 2024 design: visual encoders, language backbone, and action decoding.
- Kim and colleagues, OpenVLA: An Open-Source Vision-Language-Action Model, original 2024 paper: action discretization, training objective, and evaluation conditions.
- Black and colleagues, π₀: A Vision-Language-Action Flow Model for General Robot Control, original 2024 paper: state-conditioned action generation, flow matching, action chunks, and inference.
Continue with behavior cloning to inspect a learned action rule, or diffusion policy to study continuous action generation. For the software connections carrying observations and commands, read ROS 2 topics, services, and actions.