Robotics is ushering in its own Computer Use moment.
Yesterday, Anthropic released Fable 5.2, bringing new advances. Compared with its predecessor, its physical robot control capabilities improved markedly, enabling it even to complete tasks on which it had never previously undergone specialized training.
In the RoboDojo Sim simulation evaluation, Fable 5.2 scored 64.9%, far exceeding GPT-6 Astra (29.8%) in Anthropic's internal test under the same configuration.
As early as September, MIT researcher Phillip Isola proposed in “Robot-Use Agents” that, in the future, robots connected to the internet and exposing their sensor and actuator interfaces could all become tools that general-purpose Agents could call on.
This also aligns withthe System 1/System 2 division-of-labor approach in embodied intelligence: general-purpose models handle high-level reasoning and task planning, while low-level policies execute specific actions. Models such as GPT-6 Astra andFable 5.2 have preliminarily validated the feasibility of this approach.
However, two key issues must still be resolved for this approach to work truly end to end.
The first is the model. Closed-source frontier models have demonstrated impressive robot manipulation capabilities, but researchers cannot directly access their weights, much less use robot interaction data to continuously train them.
The second is the system. Robots differ significantly in hardware architecture, sensors, and control interfaces, and simulation environments differ from the real world as well. If a tool system had to be redeveloped every time a new robot was connected, the costs of cross-environment transfer and scaled-up experiments would remain prohibitively high.
Therefore, RUA needs a unified interaction interface that abstracts away the underlying differences between different hardware and environments, allowing models to obtain perceptual information, invoke control tools, and receive execution feedback in a consistent manner.
Recently, a research team from Tsinghua University comprising Ding Ning, Liu Zhiyuan, Xu Mengdi, Bowen Zhou, and others, together with the team of Pieter Abbeel, a UC Berkeley professor and co-director of the Berkeley Artificial Intelligence Research Lab (BAIR), jointly introduced and open-sourced uni-robot-agent, seeking to establish a unified, modular Harness for Robot-Use Agent (RUA) that enables language models to control different robots and simulation environments through standardized tool interfaces.
Paper link: https://arxiv.org/pdf/2611.01560
Blog link: https://openrua.ai/uni-robot-agent
- Code link: https://github.com/Open-RUA/uni-robot-agent
The framework performed strongly in empirical evaluations. On the RoboDojo benchmark, the open-weight Qwen3.8-Flash-Next model achieved 31.39 points with the framework, surpassing the 28.97 points for GPT-6 Astra + RoboProbe previously reported by RoboDojo. Under the same framework, GPT-6 Astra achieved a further 51.90 points.
In an evaluation of 19 tasks on real robots, Qwen, without task-specific fine-tuning, scored 53.50 points, outperforming the fine-tuned π₀.₅ and zero-shot Dream-1 by 5.75 and 10 points, respectively. GPT-6 Astra with the framework achieved 80.50 points.
Beyond the benchmark scores, the work also tested the extent to which the model could control robots, identifying the causes of failures, capability gaps, and runtime costs.
Going further, they aim to build a general-purpose foundation that enables open-source models to connect to different robots and paves the way for subsequent training and on-premises deployment.
Why is a unified interface the foundation for cross-environment generalization?
In the Coding Agent domain, the same set of tools can be used for different programming tasks. Models read files, execute code, and inspect execution results through similar interfaces, allowing them to reuse existing tool-use experience across different environments.
Robot manipulation involves greater hardware variation. Even when all tasks involve grasping objects, the action interfaces of single-arm and dual-arm robots may differ. Models must understand new parameters, coordinate systems, and control rules, making existing tool-call experience difficult to transfer directly.
uni-robot-agent aims to bridge these differences through a unified tool interface.
For example, a Franka single-arm robot must locate, grasp, and stack blocks; an ALOHA dual-arm robot must coordinate two robotic arms to hand over an object; and a real YuMi dual-arm robot must control the alignment and insertion of a pin into a hole. Despite differences in hardware structure, action space, and task objectives, the model can invoke unified perception, motion, skill, and task-control interfaces, while lower-level adapters handle the control differences among different robots.
In other words,models can follow consistent tool-calling rules. When integrating a new environment, developers primarily need to implement the corresponding adapter, reducing the need to modify the upper-layer Agent logic.
Tool reuse across different environments.
A unified interface also has a deeper significance.
If different environments use the same tool semantics, parameter units, and feedback formats, the interaction trajectories produced while robots perform tasks can more easily be combined. Experience accumulated by the model in single-arm stacking, dual-arm object handoff, and real-robot insertion tasks can be consolidated into a single data system.
This designcreates the conditions for cross-environment Agent training, enabling researchers to use interaction data from different robots to train the same model and learn transferable tool-use and manipulation capabilities.
How is uni-robot-agent designed?
The design of uni-robot-agent centers on two objectives: first, providing a unified foundation for cross-environment Agent training; and second, evaluating as objectively as possible the existing robot-manipulation capabilities of general-purpose models.
To this end, the research team divides the system into five layers: Agent, Orchestration, Tools, Services, and Backends.
The Agent layer interacts with the model and manages context and task decisions; the Orchestration layer schedules tool requests; the Tools layer provides unified operation interfaces to the model; the Services layer manages runtime resources; and the Backends layer connects to specific robots, simulators, and model implementations.
This modular structure supports independently replacing models, tools, and robot backends. Researchers can compare the performance of different models and accumulate structurally consistent interaction data without rewriting the entire system.
Objective evaluation requires conditions as close as possible to real-world deployment. Simulation environments can directly provide precise 3D object positions, CAD models, hidden scene states, and even judgments of whether a task was successful—information that typically cannot be obtained directly from real robots. If models rely on such privileged information to complete tasks, it becomes difficult to determine how much visual understanding, spatial reasoning, and manipulation capability they actually possess.
Therefore, in the main simulation experiments, uni-robot-agent provides models only with RGB images, camera calibration, robot proprioceptive state, and task instructions. It does not provide depth ground truth, object CAD assets, hidden scene annotations, or task-validator feedback, and it does not allow access to external networks. At the same time, the research team also rules out shortcuts specifically tailored to evaluation tasks, preventing models from achieving high scores through prewritten task answers.
Then, without this additional information, how can general-purpose models complete complex robot-manipulation tasks?The key lies in rationally organizing the information available to the model and ensuring reliable tool execution.
Key design choices. Each column presents two complementary designs aimed at ensuring physical-state consistency, action-execution precision, and context management.
For example, when a robot grasps an object, the model needs to calculate the grasping coordinates based on the image and the robotic arm’s position. If the image comes from a previous moment while the arm’s state has already been updated, the calculated result may contain errors.
uni-robot-agent binds camera images, calibration information, and robot state to the same moment in time, reducing localization errors caused by state desynchronization. When executing actions, it deduplicates requests to prevent repeated operations and monitors the robotic arm’s motion status to identify stalled actions or execution anomalies.
After an action is completed, the system feeds the execution result and new environmental observations back to the model, helping it determine whether the task has succeeded and whether the next operation needs to be adjusted.
To address the long execution time of robotic actions, the framework also introduces asynchronous execution. The model can continue processing other work while some tools are running, while motion requests for the same robotic arm are queued sequentially to prevent command conflicts.
These designs incorporate engineering experience from the robotics field into the Harness, reducing the interference of information misalignment, repeated actions, and missing feedback with model performance.
By organizing information sensibly and providing reliable control tools, uni-robot-agent enables general-purpose language models to demonstrate competitive robotic manipulation abilities without relying on privileged simulation information or task-specific shortcuts.
Experimental validation: How far can open models go?
To assess the practical effect of the unified Harness, the research team first conducted simulation experiments on RoboDojo.
The benchmark contains 42 robotic manipulation tasks spanning five dimensions: generalization, fine-grained manipulation, long-horizon execution, memory, and open-vocabulary instructions. Each main experimental configuration was evaluated 252 times.
The impact of the unified Harness is substantial.
RoboDojo benchmark results. Avg. denotes the unweighted average of the metrics across the five capability dimensions. †GAP (Hybrid) indicates that the approach incorporates π₀.₅.
Using the same Qwen3.8-Flash-Next model, RoboProbe achieved only 1.90 points, RPent scored 7.75 points, and GAP Hybrid, which incorporates π₀.₅, reached 21.01 points.
With uni-robot-agent, Qwen’s score increased to 31.39 points, with a success rate of 25%. Compared with RoboProbe, its overall score improved by 29.49 points.
These experiments show that,even when model weights remain unchanged, the design of the tool system can significantly affect robotic manipulation performance.
The framework was also adapted to other models. DeepSeek-V4.1-Flash achieved 27.11 points, while GPT-6 Astra reached 51.90 points with a success rate of 46.67%.
Different models have different strengths. Qwen and DeepSeek perform relatively better on memory and open-vocabulary tasks than on fine-grained manipulation and long-horizon execution; some specialized VLA and WAM models have advantages in long-horizon tasks.
Real-robot tests further evaluated the models’ capabilities in physical environments.
The research team conducted 19 manipulation evaluations using a dual-arm robot, covering five dimensions: spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks. Specific tasks included inserting test tubes into a rack, passing a bottle between two arms, stacking blocks in color order, wiping a whiteboard, and completing a Tower of Hanoi, among others.
Representative real-robot tasks across the five dimensions.
GPT-6 Astra scored 80.50 points, including a perfect score of 20 points in the compositional generalization dimension; Qwen scored 53.50 points, outperforming the task-fine-tuned π₀.₅ and zero-shot Dream-1.
Real-robot evaluation results. Each capability is worth a maximum of 20 points, and the maximum total score is 100 points.
However, this comparison has differences in input conditions. Qwen and Astra used RGB images and depth information provided by the RealSense D405 sensor, whereas π₀.₅ and Dream-1 used only RGB. Therefore, the differences in performance cannot be attributed entirely to the models themselves.
More noteworthy is which external information the models rely on.
When both depth and camera calibration were provided, Qwen scored 35.89 points. After removing depth while retaining calibration, its score was 31.39 points. After further removing calibration and retaining only the RGB images and the robot’s own state, the score dropped to 9.13 points.
Under the corresponding settings, Astra scored 55.83, 55.04, and 31.29 points, respectively.
Robotic manipulation performance with reduced geometric information. The figure records the process and final scores of the model completing 42 tasks through 771 tool calls with the skill library enabled.
This indicates that the models can leverage camera calibration, multi-view observations, and geometric computation to complete some spatial localization tasks without direct depth measurements. However, after calibration was removed, manipulation performance dropped significantly, and it remains difficult to recover reliable spatial relationships from RGB images alone.
Whether the predefined skill library affects evaluation results is also a question the research team focused on.
After disabling the skill library and skill-writing functionality, Qwen’s overall score dropped only from 31.39 to 30.31, retaining 96.6% of its original score, and its success rate fell only from 25% to 24.58%.This indicates that its performance does not primarily depend on prepackaged task skills; standardized perception and motion tools, together with the model’s own reasoning abilities, are already able to support a substantial portion of manipulation tasks.
The impact of skill guidance on task performance.
TAG193@@In terms of cost, Qwen’s average API cost per task was approximately $1.09, DeepSeek’s was $0.94, and Astra’s reached $21.29.
Execution speed also affects the efficiency of robotic agents. When the aggregate score first reached 25 points, Astra had consumed approximately 7.4 million tokens cumulatively and taken 42.6 minutes; Qwen consumed 23.5 million tokens and took 95.5 minutes; DeepSeek consumed 49.9 million tokens and took 99.2 minutes.
To reduce waiting times, the research team further tested an asynchronous pipeline.
TAG201@@For all five selected tasks, pipeline execution took less time than serial execution, with reductions ranging from 4.8% to 73.7%. Among them, the conveyor-belt grasping task was reduced from approximately 59.7 minutes to 15.7 minutes.
When do general-purpose models begin to develop robotic manipulation capabilities?TAG207@@
The unified Harness enables different models to be tested under similar interaction conditions and allows us to ask a more fundamental question:
What capabilities do general-purpose models actually need in order to complete robotic tasks?
TAG213@@The research team further compared 13 models from different families, generations, and scales and found no simple correspondence between robotic manipulation capabilities and model size or release date.
Robotic manipulation performance across different generations, model families, and parameter scales.
Taking the Qwen series as an example, none of the tested Qwen3.5, 3.6, and 3.7 models recorded a complete task success. With Qwen3.8, the 27B model achieved a success rate of 8.06%, while Flash-Next improved it to 25%. Although the 27B model has fewer parameters, it was able to complete some tasks that older large models could not.
The GPT series also shows generational changes. GPT-5.6 Sol can already complete some robotic tasks, with a success rate of 19.17%. GPT-6.1 Sol improved this to 37.50%, and GPT-6 Astra further increased it to 46.67%.
These results suggest that robotic manipulation capabilities may have long existed in general-purpose models, and that their improvement may be related to training-data distributions, combinations of capabilities, and model training methods.
By analyzing failure trajectories from different models, the team identified four key capabilities.
The figure shows the correlation between each model’s aggregate score on RoboDojo and its error rate across the following four capabilities. The lower the error rate in these four dimensions, the better the model’s performance on robotic tasks generally.
First is visual understanding . Robots need to recognize objects, determine their orientation, and verify the state after an operation. Some earlier Qwen models mistook the numeral 0 for 9 or judged an upside-down cup to be upright, causing subsequent actions to be based on incorrect observations.
Among the trajectories inspected, this type of error accounted for 12.2% to 23.8% in some older models. No such errors were found in the corresponding inspected samples for Qwen3.8-Flash-Next.
Second is spatial localization . Models need to determine a target’s three-dimensional position from images. When depth information is unavailable, multiple viewpoints can be used for triangulation, but matching the same physical location across viewpoints remains difficult.
The paper found that even for GPT-6.1 Sol, roughly 31.5% of the relevant triangulation calls still failed to pass geometric consistency checks.
The third category is geometric calculation . Robots need to convert positions and orientations between the camera, world, and robotic-arm coordinate frames. In some older Qwen models, the error rate in these calculations reached 47.3% to 81.8%. Qwen3.8-Flash-Next reduced it to 8.2%, and DeepSeek-V4.1-Flash further reduced it to 1.5%.
Finally, there is embodied reasoning . Models must also determine whether a grasp pose is reasonable, whether the robotic arm can reach the target, and whether an action poses a collision risk. Even after locating the target, some models repeatedly fail to grasp it because of incorrect wrist poses.
The research team also conducted correlation analyses between robotic manipulation performance and general-purpose benchmarks such as ERQA, MMMU-Pro, ScreenSpot-Pro, and LiveCodeBench v6. They found varying degrees of correlation between visual understanding, embodied reasoning, spatial localization, coding ability, and robotic performance. Therefore, the ability to operate robots may be the result of generalizing a combination of foundational capabilities from different domains .
The relationship between robotic manipulation performance and general-purpose model capabilities.
Robotic manipulation involves continuous visual judgment, spatial localization, geometric calculation, and physical decision-making. An error at any stage may cause the task to fail. Future improvements to RUA capabilities will require simultaneous attention to these interconnected foundational capabilities.
General-purpose framework: what does uni-robot-agent support?
In addition to directly invoking tools, uni-robot-agent also supports Code as Policy, Hybrid mode, where VLMs collaborate with VLAs/WAMs, and Clean Mode, which retains only basic tools.Different approaches can be tested within the same framework.
Code as Policy enables models to control robots by writing code. The model can perform calculations in the program, invoke actions, and decide whether to continue based on execution feedback.
The research team used Qwen to compare code-based control with direct tool invocation across five capability dimensions of RoboDojo. Across 252 RoboDojo evaluations, Qwen scored 25.60. Compared with direct tool invocation, code-based control achieved a higher score in the fine-grained manipulation dimension, reaching 17.40, although its overall performance still lagged.
Score comparison for Qwen in Code as Policy mode.
The framework also supports a Hybrid mode that enables VLM and VLA to collaborate on tasks. The VLM is responsible for understanding instructions, analyzing scenes, and organizing tasks, while the VLA generates action chunks based on observations, with execution feedback informing subsequent decisions.
The team demonstrated this collaboration in simulated environments for platforms including a Panda robotic arm equipped with an Allegro dexterous hand, the Unitree Go2 quadruped robot, the Unitree G1 humanoid robot, and the R1Pro wheeled robot. They also integrated action models such as π₀.₅, GR00T, and NaVILA, covering dexterous manipulation, humanoid robot operation, and mobile navigation.
VLM and VLA collaborate in different environments through uni-robot-agent.
As model capabilities improve, some computational and operational logic currently encapsulated in tools can gradually be handed over to the models themselves, and the Harness is expected to become thinner accordingly. To explore this direction, uni-robot-agent provides Clean Mode, retaining only four basic tools—move, hold, finish, and capture_scene—and allowing the model to complete tasks through observation, computation, and action composition.
In a camera-calibrated setup, Qwen and Astra in Clean Mode retained 90.4% and 90.5% of their scores with the full tool configuration, while average token usage was reduced by 35.7% and 78.8%, respectively. Comparisons with previous full configurations show that models can maintain most of their task performance after tools and context are streamlined, providing preliminary evidence for thinning the Harness and improving token efficiency.
Token consumption and score comparison for Qwen and Astra in Clean Mode.
The framework’s compatibility has also expanded to more physical robots.
The team integrated physical robots such as ARX One and AgileX Piper-X. Differences in how different hardware is controlled are handled by low-level adapters, while models continue to observe their environment and execute actions through a unified interface, adjusting their next steps based on feedback. In addition, AgiBot G2 and Unitree G1 are also being integrated.
Physical robot integration with ARX One and AgileX Piper-X.
Integration code and deployment instructions are also available, allowing developers to run uni-robot-agent on the corresponding hardware, swap out models, and test their own tasks.
Conclusion
Robot intelligence stands at a new crossroads.
The traditional specialized-policy approach is tightly coupled to hardware, is expensive to scale, and faces challenges in generalization and long-horizon tasks. The uni-robot-agent approach instead allows general-purpose models to control robots directly through a unified interface, resulting in a markedly different scaling model.
uni-robot-agent is the first open-source release from the OpenRUA project.OpenRUA was initiated by a group of researchers from leading universities and cutting-edge laboratories. The team’s expertise spans embodied intelligence and language models, with the goal of building open infrastructure for agents capable of acting in the physical world.
Team homepage: https://openrua.ai/
From this perspective, the significance of uni-robot-agent lies not only in providing a robotic manipulation framework, but also in demonstrating that, with well-organized information and reliable tool feedback, open-weight models can achieve competitive performance in both simulations and on real-world robots.
However, this is only the first step. The key to what comes next is whether the model can be continuously updated using interaction data. uni-robot-agent has already laid the groundwork, while subsequent training and evolution have only just begun.
Reference links:
https://www.anthropic.com/research/project-pilot
https://www.anthropic.com/research/claude-plays-robotics
https://x.com/FredaDuan/status/2116727410972590380
