Robotics Briefing /
What does the robot actually see?
When Simon Cowling dimmed the room around a robot arm, he measured an elevenfold drop in light. Anyone watching through the robot’s front camera would have barely noticed. That gap matters when judging what a robot can do. It also runs through a new exam for robot software, where giving a controller access to information it could never sense on a real machine changed the score.
Your quick takeaways
- A robot arm put the right ball in a basket in 20 of 40 attempts under normal lighting and 19 of 40 in dim light. Automatic exposure likely helped. The front camera’s recorded image dimmed by only 3%.
- Across 391 simulated tasks, the strongest coding agent solved 30% under standard sensor and reset rules. With access to the simulator’s exact state and unrestricted resets, it solved 77%.
- NTT reports that a small robot loaded a shelf successfully in 92.3% of tests when it could compare predicted outcomes before moving, up from 73.1% without that check. Airport baggage trials are still ahead.
Papers
Turning down the lights changed almost nothing
Simon Cowling at fysical.ai · Catch-up from November 10 · Preprint · 80 physical trials on a low-cost arm
People who test robots often record how bright the room was, measured in lux. The assumption is that the same light gives the camera the same picture. Simon Cowling tested that on a common hobbyist setup, an SO-101 arm with two USB cameras. Cowling trained four widely used controllers on the same 50 demonstrations of one task: pick up a green ball and put it in a basket while ignoring two red ones. Each controller then got ten attempts at about 520 lux and ten more at about 45 lux, with the artificial lights off and the windows screened.
Success was 20 of 40 in normal light and 19 of 40 in low light. No controller changed by more than one success in ten. The video shows how little the camera input changed. Room light fell more than tenfold, yet the front camera’s images were 3% darker on average and the wrist camera’s 19% darker. Both cameras were set to automatic exposure and white balance. In color, the normal-light images were further from the training recordings than the dim ones were.
The room got darker, but the recorded images changed little. The author checked the camera settings afterward and did not repeat the test with exposure locked. With one arm, one task and ten attempts per controller in each condition, a meaningful drop in performance could easily go undetected. Readers can inspect the trial recordings and analysis code in the public dataset.
An exam for AI agents that write their own robot programs
Kangrui Wang, Manling Li and colleagues at Northwestern, Carnegie Mellon, UC Berkeley, Stanford and other universities · Catch-up: November 9 preprint · Simulation, with one small physical demonstration
Coding agents are AI systems that write and run software on their own. Researchers now ask whether one can control a robot directly. The agent would read the documentation, look through the cameras, and write a controller or even train one. The Embodied First Exam tests that. It packages 1,156 simulated tasks for 47 robots, from household chores and rope handling to humanoid sports. A checker the agent cannot reach grades each attempt.
A task can run under different rules. In the standard setting the agent gets only what the robot’s sensors report and usually cannot reset the scene after a mistake. In the privileged setting it can read exact object positions from the simulator and retry as often as it likes. On a core set of 456 hard tasks, the best of eight agents, GPT-6 Astra running in the Codex tool, solved 33.6% under standard rules. On the 391 tasks that have both versions it solved 30% standard and 77% privileged. A controlled comparison suggests that the freedom to retry matters more than perfect knowledge. Exact positions alone raised success from 28% to 45%, and resets alone raised it to 53%.
Each agent made one attempt per test, so luck and ability are hard to separate. The core set leaves out tasks that an earlier model from the top scorer’s family could solve, and the authors say this may tilt the comparison. AI agents wrote the labels for where attempts failed, with human oversight. The only physical test was a buzz-wire game with one arm, two agents and eight tries each, and it is kept out of the scores. The code and recordings of the runs are public. The code page notes that some of the underlying simulators have to be set up separately.
A hundred household chores, in simulation
Yelin He and colleagues at Intellifusion · Catch-up from November 10 · Technical report · Simulation only
The BEHAVIOR Challenge is run from Stanford and tied to this week’s Conference on Robot Learning. It asks one simulated robot with a mobile base and two arms to complete 100 household activities, from turning on a radio to cooking a meal. Entrants get 200 human-steered demonstrations of each. A run counts as a full success only if every goal condition holds at the end. Otherwise it earns partial credit.
The second-place team built on Cosmos 3, a model trained to predict upcoming video frames along with the robot’s movements. They added a separate estimator of how far along a task is. They also trained small add-on modules for picking, placing, opening and closing. On the organizers’ private test, with 20 setups per task, the system fully completed 41.2% of runs. The team says it used no closed commercial models and no coding agents.
Everything here happened in a simulator, and no physical robot was tested. The report compares whole configurations and says it does not isolate what each addition contributed. It links no code or model files.
News
Checking whether a stack will fall before placing the next item
NTT · November 11 announcement · Company-reported physical test
Two places on a shelf can look equally inviting until a new item shifts the load underneath it. NTT built a system that compares the moves a robot is considering. For each move, it estimates how objects will shift using properties such as mass and orientation, then uses those estimates to generate video of the expected outcome. An existing controller can use the predictions to choose a placement less likely to cause a fall.
NTT reports that success on a shelf-loading task with a small robot rose from 73.1% to 92.3%. The announcement gives neither the trial count nor the range of loads tested. Baggage varies in weight even among bags of similar size, which makes it a practical test of the method. NTT and Mujin plan field trials. The shelf result is the evidence available so far.
Highlights
A ninth-place entry you can download
Robot Learning Collective · November 10 technical report · Simulation only · Scores from the team’s own runs
Last year’s winners of the same contest finished ninth this time and released their code, a single model for all 100 tasks, and an account of where it failed. Their trained controller alone averaged 0.392 on the contest’s partial-credit score and scored zero on eight tasks. For 14 tasks it handled poorly, the team let a general-purpose coding agent drive the robot through software tools after practicing in the simulator. On those tasks the agent averaged 0.639 against 0.108 for the trained controller. The team says the agent was imprecise at pouring and placing and sometimes looped when a task looked finished. The organizers re-tested only the top entries, so these scores are the team’s own.
Read this first
For a sense of where simulated household robots stand, open the Robot Learning Collective’s page for its BEHAVIOR entry. It reproduces a results table for the top ten teams, and the leading entry there is listed at 48.2% full-task success. The page also carries evaluation videos. A long chore earning partial credit looks very different from a finished one. The table is the team’s copy of the standings and not the official leaderboard.
Read the paper