Robotics Briefing /
Can a robot borrow someone else’s practice?
Recording a person steering a robot through a task, one attempt at a time, is slow. Much of the newest work asks how little of that a robot needs if it can lean on experience gathered elsewhere: people filmed doing chores, another robot’s recordings, or motion produced by software. NVIDIA’s new model made the largest claim this week. Two smaller studies show where borrowing helps and where it runs out. A third asks how a robot stays quick as the model guiding it grows.
Your quick takeaways
- NVIDIA reports 40% success across 50 tasks with no demonstrations from its two-armed test robot, and 95% after five demonstrations per task. The researchers selected tasks that had already reached at least 80% success with five demonstrations.
- A University of Pennsylvania method raised an unchanged robot policy from 20% to 80% success across five physical tasks by steering it with recordings made in other labs, with different cameras and objects. Each task got ten attempts.
- An XPENG Robotics humanoid placed objects correctly in 23 of 24 trials when its movements no longer waited for its slower language-and-vision model. The version that waited managed 16.
Papers
Following a recording made somewhere else
Subin Kim, Anh-Quan Pham and Dinesh Jayaraman · November 7 preprint · Physical tests and simulation
General-purpose robot policies often stumble when the camera moves or the objects change. RAGu leaves the policy’s training untouched. It searches a large public collection of robot recordings for the same kind of task, such as opening a drawer. It reshapes each recorded hand path to fit where the objects sit in the new scene. Then it picks the path that asks the policy to change its own plan the least, and nudges the policy toward that path as it acts.
On a physical arm, the unassisted policy succeeded in 20% of attempts across five tasks. With RAGu it reached 80%, including all ten attempts at wiping a table. A method that points the policy only at goal locations reached 40%, and often lifted the towel off the table between points. In simulation, RAGu raised average success across nine tasks from 23.6% to 36.4%.
Ten attempts per task is a small sample. Reshaping the recordings also failed about a third of the time in the simulated tests, mostly because the source recording was unusable. The method needs a library that already contains the right kind of task, and it can only adjust movements the policy is able to make.
A bigger model without slower hands
Bingzhang Wang and colleagues at XPENG Robotics and the University of Washington · November 9 preprint · Simulation and physical humanoid tests
Many robot policies pair a large model that reads the camera image and the instruction with a smaller part that produces movements. If the small part waits for the large one, the robot acts on an image that is already old. DASH runs the two as separate programs. The movement part looks at a fresh camera image through its own small vision network and reuses the large model’s most recent output. During training it is deliberately given out-of-date output, so that stale information is familiar when it meets it on the robot.
In a simulator modified so that thinking time costs real time, the waiting design slowed from 9.7 to 4.1 decisions per second as the large model grew tenfold. DASH held 20 per second at every size, and success rose by one to six percentage points over 250 episodes per setting. Separation alone did not help. Without the stale-input training, results were slightly worse than waiting.
On a humanoid with multi-fingered hands, DASH completed 23 of 24 pick-and-place trials and the waiting version 16. The two physical setups also blended overlapping movement plans differently, so the gap cannot be credited to one change. The objects sat still, and the team names moving objects and longer tasks as the next test.
Practice generated by software, and a coach for the mistakes
Kevin Park, Izat Rasyad, Dvij Kalaria and Shankar Sastry · November 7 arXiv posting · Accepted at two CoRL workshops · Simulation only
DreamVLA tries to train a humanoid without anyone steering it through the task. A motion generator produces human-like movements for an instruction. A controller trained by trial and error in simulation, with access to exact object positions, turns those movements into successful attempts. A camera-based policy then learns from the recordings. When that policy fails, the controller takes over from a moment inside the failed attempt and shows how to finish.
Those corrections mattered more than volume. With 800 ordinary recordings, the simulated Unitree G1 never managed to pick an object from a platform in 100 tries. With 800 recordings that included three rounds of corrections, it succeeded in 54. Drawer opening rose from 54% to 60% and button pressing from 57% to 69%.
Walking to the platform and then picking up the object reached only 18%, and the controller that supplied the corrections itself succeeded about three times in four on that task. Nothing was tested on a physical robot, and the simulated camera images were not varied to prepare for one.
News
NVIDIA describes GR00T 2, with the code still to come
NVIDIA · November 9 CoRL announcement · Company-reported physical tests
GR00T 2 predicts a short stretch of future video together with the movements that would produce it. NVIDIA says it trained the model on about 400,000 hours of video filmed from people’s point of view, about 10,000 hours from handheld grippers and 23,500 hours of robots steered by operators.
The headline test used a two-armed robot with multi-fingered hands and 50 tasks it had no recordings of, though they appeared in the human video or on another robot. The company reports 40% success with no demonstrations on that robot, 71% with one per task and 95% with five. The whitepaper notes that the 50 tasks were selected from those the model already learned from five demonstrations, so the last figure is high partly by construction. It does not state how many trials stand behind each number.
NVIDIA ran all of these tests itself. Its project page lists the code as available in December, and the page does not link model weights. A smaller variant meant to run on a humanoid’s onboard computer scored about half as well on the hardest hand tasks, 34.3% against 68.6%.
Highlights
What can be downloaded today
NVIDIA resources released with GR00T 2
Two pieces of the project are already public. A 100,000-hour portion of the human video collection is posted under a license that allows reuse with credit. NVIDIA has also published the parts lists, design files and assembly guides for the handheld gripper, the hand-worn recording device and the three-jointed robot neck it used while gathering data. The model itself is the piece still missing.
Read this first
Find the limitations NVIDIA lists in the GR00T 2 whitepaper before reading the results. The company writes that up to 20 trials per task can show large gaps between systems but not small ones, and that some failures were scored as successes by the system’s own checks. Then return to the charts and see which comparisons still look secure.
Read the paper