explainer
Robot autonomy and teleoperation: who makes each decision?
Compare direct teleoperation, supervisory control, shared autonomy, and bounded task autonomy. Follow one grasping task, calculate communication delay, and read intervention records.
What you will learn
- Identify who selects goals, directs motion, and handles recovery in a robot task.
- Distinguish direct teleoperation, supervisory control, shared autonomy, and bounded task autonomy.
- Explain why a human-operated robot can still use local feedback controllers.
- Calculate observation-to-command delay and interpret intervention records.
Before you start
- A robot’s sensing-to-action loop
- Basic arithmetic with seconds and meters
A robot’s motion alone cannot tell you who made its decisions. An arm can pick up a cup while a person steers every movement, approves each step, or watches a robot carry out an assigned task. To describe its autonomy, trace who chooses the goal, commands the movement, and handles failure.
Describe autonomy through decisions
Teleoperation lets a person operate a robot through an interface. The interface might send joint positions, tool velocities, or higher-level requests. Autonomy describes decisions and actions the robot carries out within an assigned scope.
ESA’s Robot-User Interfacing overview describes a range of human involvement. Its interactive-autonomy approach divides work into subtasks the robot handles and subtasks that require the user. That allocation provides a useful starting question: who does which part?
This lesson uses four functional descriptions: direct teleoperation, supervisory control, shared autonomy, and bounded task autonomy. Researchers and applications draw their boundaries differently. These examples provide no universal level scale or ranking of overall capability.
Compare four ways to move one cup
Imagine an arm moving a cup from a table to a tray. The camera can see both objects, and the robot has joint encoders and a gripper. We will change the division of work while keeping the same task.
The physical-AI lesson separates perception, estimation, planning, and control. Here the extra question is which decisions a person supplies. These drawings describe possible arrangements; they report no robot experiment.
The table makes each allocation explicit. Monitoring can run during every stage, even though the diagram lists it as a separate job.
| Arrangement in this example | Perceive | Select goal | Plan and execute | Monitor and recover |
|---|---|---|---|---|
| Direct teleoperation | Human reads the camera view | Human chooses the cup and tray | Human steers; robot tracks commands | Human checks progress and chooses retries |
| Supervisory control | Robot estimates the cup pose | Human chooses the cup and tray | Robot plans a move; human approves it; robot executes | Robot reports; human authorizes the next step or recovery |
| Shared autonomy | Robot estimates the scene; human watches | Human intends the cup and tray | Human steering and robot assistance both shape motion | Both monitor; human can override under the stated rule |
| Bounded task autonomy | Robot estimates object and tray poses | Human assigns the task; robot selects intermediate steps | Robot plans and executes the movements | Robot uses defined recovery rules, then requests help |
Real systems can switch arrangements during a task. A robot might handle the approach, ask a person to guide a difficult grasp, then resume its placement routine. Record those changes when describing the result.
Keep local feedback in the picture
A person can command a tool velocity while the robot closes its own joint-control loops. The person chooses the requested movement. The controller compares measured motion with that request and updates actuator commands.
Modern Robotics’ control overview shows desired motion entering a controller alongside sensor feedback. That structure also fits a human-supplied reference. Direct teleoperation can therefore include substantial local control without delegating the task goal or route.
For example, a joystick might request 0.10 m/s along the table. A local controller works to produce that tool motion. Automatic joint tracking alone does not establish that the robot can choose a cup, detect a failed grasp, or decide what to try next.
The feedback-control lesson isolates measured error and corrective effort. It helps distinguish a motor-level correction from a task-level decision.
Separate supervision from shared control
In our supervisory-control example, the person approves a move and lets the robot execute it. The robot reports completion or a problem. The person then authorizes the next step.
In our shared-autonomy example, human input and robot assistance both affect the current motion. The person steers toward the cup while assistance adjusts the approach. A system must define how it combines those inputs and how the person can disagree.
Javdani, Srinivasa, and Bagnell’s 2015 shared-autonomy paper studies assistance when the robot does not know the user’s intended goal. Their method reasons over possible goals and selects assistance under that uncertainty. It provides one concrete approach to shared autonomy, not a required algorithm for every shared-control system.
Their study found quicker completion with less user input compared with a predict-then-blend method. Users gave mixed ratings and raised concerns about control authority. Those findings belong to that study; task speed alone cannot describe how an operator experiences assistance.
Command transport answers a separate question. ROS 2 topics, services, and actions can carry streams, requests, or goals with progress reports. An action interface supports a goal exchange, but its existence does not establish who chose the goal or how much help execution required.
State the task and operating conditions
For the cup example, bounded task autonomy means the robot handles the agreed task after assignment. It chooses intermediate movements and evaluates their outcomes. A person can still choose the overall task and remain available for exceptions.
Make that scope concrete before evaluating it:
- Which cups, trays, starting poses, and lighting conditions belong in the test?
- What counts as success, including placement tolerance and time allowed?
- Which failures can the robot detect and recover from on its own?
- When must it stop, request help, or return control to the operator?
A routine might allow one grasp retry after a detected empty gripper. An unfamiliar object or a blocked tray might trigger a help request. That description tells the reader more than a broad claim that the arm “works autonomously.”
Successful cup placement within those conditions supports that task claim. It does not establish the same capability with slippery objects, cluttered shelves, or a different gripper. Moravec’s paradox explains why familiar physical tasks can conceal difficult perception and movement problems.
Calculate a delayed remote correction
Now consider direct teleoperation during the approach. At time zero, the robot captures an image that prompts a motion correction. Assume a shared clock, fixed delays, and no additional buffering or packet loss.
The image takes 120 ms to reach the operator. After seeing it, the person takes 180 ms to send a new command. The command takes 120 ms to reach the robot, followed by 20 ms of processing before it takes effect.
| Time after capture | Event | Added delay |
|---|---|---|
| 0 ms | Robot captures the image | Start |
| 120 ms | Operator sees that image | 120 ms return path |
| 300 ms | Operator sends the correction | 180 ms human response |
| 420 ms | Robot receives the correction | 120 ms command path |
| 440 ms | New command takes effect | 20 ms robot processing |
The network round-trip delay is the sum of the two network legs: 120 + 120 = 240 ms. The observation-to-command delay also includes the person and the robot’s processing:
total delay = 120 + 180 + 120 + 20 = 440 ms = 0.440 s
travel before the new command
= constant speed × total delay
= 0.10 m/s × 0.440 s
= 0.044 m = 44 mm
For this calculation, assume the tool keeps moving straight at exactly 0.10 m/s until the new command takes effect. The 44 mm is travel before that change. Deceleration and later motion require another model, so this number gives no stopping distance or safe clearance.
The robot’s local feedback loop can keep running during the delay. A local detector could also trigger its own response, if the system provides that function. This timeline includes only the remote correction and establishes no bound for a complete safety system.
Record intervention and recovery
An autonomy claim needs the full set of task attempts. Record human inputs during operation, including small corrections that a short video may hide. Separate setup before a trial from assistance during it.
A useful trial record includes:
- Initial conditions, assigned task, and the success rule.
- Human commands and changes in who controls each part of the task.
- Each help request or intervention, its reason, and its duration.
- Robot retries, failures, completion time, and final outcome.
Count completion with assistance separately from completion without assistance. Keep failed attempts in the denominator. Report intervention frequency alongside task difficulty and the rule that triggered intervention.
An interval with no operator input establishes only what happened during that interval. Stronger evidence requires repeated trials under stated conditions. A robot can also act autonomously and fail; the allocation of decisions and the quality of those decisions are separate measurements.
Separate learning from operating authority
Teleoperation can produce demonstrations for behavior cloning. During data collection, the operator supplies the actions. A trained policy may later supply actions itself, but that later behavior needs its own rollout evaluation.
An instruction such as “put the cup on the tray” specifies an interface to a task. It does not reveal whether a person guides the arm behind the scenes, approves intermediate steps, or corrects mistakes. The vision-language-action lesson follows how observations and language can enter an action policy.
A learned policy can take task decisions, and a programmed planner can do so too. The use of machine learning alone does not determine the robot’s operating authority. Identify the actual inputs, interventions, and decisions when evaluating either system.
Try it yourself
Exercise 1: Use asymmetric delays. An image takes 80 ms to reach the operator. Human response takes 140 ms, the command path takes 200 ms, and robot processing takes 30 ms. At a constant 0.10 m/s, what are the network round trip, observation-to-command delay, and travel before the new command?
Check the timing and distance
The network legs sum to 80 + 200 = 280 ms. Adding human response and robot processing gives 80 + 140 + 200 + 30 = 450 ms. The tool travels 0.10 × 0.450 = 0.045 m, or 45 mm, before the new command takes effect.
The event times are 0, 80, 220, 420, and 450 ms. Unequal one-way delays still add to the network round trip. This calculation again excludes braking after the command begins.
Exercise 2: Read the denominator. A person assigns the same cup-placement task for 20 trials. The robot completes 14 without help; four more complete after joystick corrections; two fail without any correction. Compute completion without assistance, completion with or without assistance, and the fraction of trials with an intervention.
Check the counts and the claim
Completion without assistance is 14/20 = 70%. Completion allowing assistance is 18/20 = 90%. An intervention occurs in 4/20 = 20% of trials.
Report all three alongside the task conditions and success rule. The corrected trials include human-directed motion, and the two failures remain in every denominator. The 70% figure measures unassisted completion in this test; it is not a universal “autonomy percentage.”
Sources and further study
- ESA: Robot-User Interfacing describes teleoperation and a division of subtasks between autonomous execution and user interaction. The four cup-task arrangements here are original teaching examples.
- Javdani, Srinivasa, and Bagnell: Shared Autonomy via Hindsight Optimization, 2015 models uncertain user intent and evaluates one assistance method against predict-then-blend control. Its reported task and preference results have the scope of that study.
- Lynch and Park: Control System Overview explains desired motion, sensor feedback, and actuator commands. The latency values and travel calculations in this lesson are invented examples with explicit assumptions.
For your next robot demonstration, write down the intervention rule before the first trial. That one line makes the result easier to interpret.