Synced Report
The success or failure of hanging a mug on a cup rack is often foreshadowed at the first point of contact.
If the handle is grasped, the mug may be stable, but the hook has no clearance to enter; after the mug body is grasped, the handle must also be properly aligned with the hook. Pouring water, opening a cabinet door, and placing a cup into a coffee machine also have their own contact requirements.
A suitable grasp must leave enough room for subsequent actions to complete the task.
Such requirements are related to the affordance of objects (affordance): for the current task, which region of an object can support which interaction. Humans select contact locations based on the goal, and the process of generating robotic manipulation data likewise requires this kind of judgment.
Figure 1. Dynamic comparison of placement and hanging tasks. The four result windows in the figure play back the execution processes of generic grasping and AffordSim, respectively; task-relevant functional regions constrain both grasp selection and the alignment of subsequent operations.
To address this issue, a team led by Professor Xuguang Lan from the Institute of Artificial Intelligence and Robotics at Xi’an Jiaotong University and the National Key Laboratory of Human-Machine Hybrid Augmented Intelligence proposed AffordSim, which incorporates open-vocabulary 3D affordance prediction into simulation-based trajectory generation. Starting from natural-language descriptions, the system sets up the scene and task, locates functional regions on object surfaces, and then combines motion planning to generate executable robotic manipulation demonstrations.
Paper: https://arxiv.org/abs/2604.11674
Project page: https://affordsim-platform.github.io/AffordSim.io/
Code repository: https://github.com/AffordSim-Platform/AffordSim
The study also developed an evaluation benchmark covering 50 tasks, seven categories of manipulation skills, and five robotic arms, supported by an asset library containing more than 500 rigid-body and articulated objects. For tasks requiring precise selection of functional regions, AffordSim reduces the need to manually specify contact locations on a per-object basis and provides an experimental platform for evaluating how policies leverage these data.
Background and Challenges
To prepare manipulation demonstrations for robots, the first step is to determine how the robot should contact the object. Existing simulation data generation pipelines commonly use one of two approaches: manually adjusting grasp positions for different objects and tasks, or invoking a generic grasp estimator to select candidate poses based on grasp stability.
Manual adjustment can encode detailed task expertise, but adding each new object or interaction may require further adjustment. A generic grasp estimator reduces this effort, but it may not account for what must be done after the object is grasped. For example, a contact position suitable for transporting a cup may impede subsequently hanging it on a cup rack.
Adding semantic-region filtering to generic grasp results can eliminate some unsuitable candidates. However, if the estimator does not generate any candidates in the target region to begin with, subsequent filtering cannot recover the missing grasps. AffordSim therefore places functional-region prediction before candidate generation, allowing task requirements to directly determine where the robot attempts to contact the object.
Contribution 1: From Task Descriptions to 3D Functional Regions and Executable Trajectories
AffordSim connects scene organization, affordance prediction, and motion planning.
First, a vision-language model parses the scene and task description separately. Scene parsing extracts objects, attributes, and spatial relations, selects objects from a USD asset library, and assembles the scene; task parsing decomposes the goal into manipulation steps and generates corresponding functional-region queries. For example, in a pouring task, the system distinguishes between the container being held and the container receiving the material, generating queries for the different interaction roles.
Next, an open-vocabulary 3D affordance model receives the object's point cloud and the query, and predicts which surface points are relevant to the current interaction. The point clouds used here accompany the simulation assets, allowing the system to constrain contact locations using 3D geometry.
Figure 2. AffordSim's data generation pipeline. The language description is parsed into a scene and a task, which are converted into functional-region queries; the 3D predictions constrain grasp candidates, and motion planning further checks whether the candidates are reachable and collision-free.
Grasp selection is divided into two stages. The first stage determines where to attempt the grasp: the system extracts representative contact points from the predicted region and combines them with local surface normals to generate candidate poses with different approach directions. The second stage determines which candidate the robotic arm can execute: cuRobo is invoked to plan a collision-free approach trajectory for each candidate, which is then scored according to joint-space path length.
The scoring rule is straightforward: if planning succeeds without collisions, the candidate receives a score equal to the reciprocal of the path length; if no feasible path can be found, its score is zero. The system selects the highest-scoring candidate, reuses its corresponding approach trajectory, and then connects it with gripper closing, lifting, and subsequent task actions to form a complete demonstration.
This design checks both functional suitability and the feasibility of execution by the robotic arm, enabling the predicted functional regions to contribute directly to trajectory generation.
Contribution 2: Evaluating Affordance-Aware Manipulation with 50 Tasks
AffordSim’s benchmark covers seven categories of skills: grasping and placing, switching, pushing and pulling, hanging, pouring, stacking, and long-horizon compositional skills. The tasks can be run on five robot arms: Franka, Kinova, Sawyer, UR5e, and xArm7.
Figure 3. Dynamic demonstrations of 50 tasks, covering diverse object interactions, manipulation sequences, and robot arms, for evaluating trajectory collection and policy execution.
The benchmark is further divided into three difficulty levels based on functional-region constraints and manipulation-sequence requirements: Easy contains 10 tasks with relatively few restrictions on contact regions; Medium contains 30 tasks that require selecting contact regions according to the function of the object; and Hard contains 10 tasks that further introduce long-horizon dependencies or stricter alignment and stability requirements.
This division enables experiments to observe how the gap among general-purpose grasping, human expertise, and affordance prediction changes as tasks increasingly depend on the correct contact location. For multi-stage tasks, the benchmark also reports the success rate at each stage to identify whether failures occur during grasping, alignment, or subsequent actions.
Contribution 3: Replacing real-scene backgrounds and supporting camera-viewpoint switching
Once trajectories can be executed in simulation, the differences between training images and real environments must also be addressed. AffordSim focuses its domain randomization on real-scene background replacement and cross-camera-view rendering: while retaining the robot and manipulated objects, it replaces the simulated background with reconstructions of real environments such as a coffee corner, library, and factory workbench, providing manipulation demonstrations with diverse environmental appearances.
The system starts with 10–20 photographs of each scene, uses Depth Anything 3 to recover depth and camera poses, and then constructs a 3D Gaussian Splatting representation.
The background can be switched along with the camera viewpoint: within the reconstruction’s coverage, the observation position and direction of the virtual camera are changed, the background is rendered again using the corresponding camera parameters, and it is composited with the simulated robot and manipulated objects from the same viewpoint. This allows the same manipulation process to be observed from multiple camera viewpoints, with the background’s perspective changing accordingly.
Figure 4. Joint variation of real backgrounds and camera viewpoints. The three rows show a coffee corner, library, and factory workbench, respectively, while the three columns correspond to camera viewpoints of −90°, −60°, and −30°. The images in each row come from synchronized camera frames of the same manipulation trajectory and are composited with the reconstructed background from the corresponding viewpoint.
In addition to background replacement, the system supports randomization of object poses, surface textures, lighting, and image noise, enabling the training data to cover variations in the environment, viewpoint, and object appearance. The study further uses these simulated demonstrations for zero-shot deployment on real robots to examine the transferability of the generated data.
Experimental Highlights
The stronger the functional-region constraints, the more pronounced the benefits of affordance guidance.In the trajectory collection experiment on 50 tasks, all methods used the same scenes and motion-planning setup. Each task was run 100 times, and the object’s initial pose was randomized within a 30 cm horizontal-position window and a 60° yaw range. The table below presents the average trajectory collection success rates reported by the paper at each difficulty level.
Table 1. Trajectory collection success rate. Manual adjustment refers to setting contact positions and orientation rules for objects; AnyGrasp+Filter adds task-relevant region filtering to generic grasp candidates. The metric here is whether demonstration trajectories can be successfully collected, not the execution success rate of a learned policy.
On Medium and Hard, AffordSim improves over AnyGrasp+Filter by 34 and 48 percentage points, respectively. Relative to the manual-adjustment baseline, it reaches approximately 93% and 89% of the baseline’s trajectory collection success-rate levels, corresponding to 79%/85% and 64%/72%, respectively. These two relative ratios reflect the gap between automatically predicting functional regions and manually adjusting each object.
The gains are observable across all five robot arms. In the cross-embodiment experiment, AffordSim achieves an average trajectory collection success rate of 76%–83% across the 50 tasks, outperforming AnyGrasp+Filter by 32–33 percentage points on every robot arm.
The collected data can be used for policy training, although complex manipulation still has room for improvement. The study compares BC, Diffusion Policy, ACT, VLA-adapter, and π0.5 on 20 representative tasks, using 300 demonstrations per task. The overall simulated execution success rates of π0.5 and VLA-adapter are 55% and 54%, respectively; on Hard tasks, they are 24% and 23%, respectively. Stage-wise analysis shows that subsequent precise alignment and long action sequences remain significant challenges.
A real-robot experiment validates the feasibility of zero-shot transfer. The study deploys a policy trained exclusively on AffordSim manipulation demonstrations to a physical Franka FR3, covering two placement tasks, one pouring task, and one hanging task. Each task is tested 10 times, without fine-tuning on real-robot data.
Figure 5. The four physical Franka FR3 tasks in the paper, arranged in a 2×2 layout: placing a banana on a plate (top left), placing a cup on a shelf (top right), pouring the contents of a spoon into a bowl (bottom left), and hanging a mug (bottom right). After being trained on simulated demonstrations, the policy is deployed directly.
Table 2. Zero-shot simulation-to-real transfer success rate. Each task is tested 10 times; VLA-adapter’s exact mean is 22.5%, which the paper’s table reports as the integer 23%.
The real-robot results show that simulation data can support direct transfer for these tasks while also revealing clear differences in difficulty: tasks involving precise hooking remain more challenging. Together with the simulation experiments, the results enable further tracing of the respective contributions of functional-region prediction, post-grasp alignment, and long-horizon execution to the errors.
Research Significance and Future Directions
AffordSim treats task-relevant contact regions as explicit conditions in data generation, linking natural-language goals, 3D geometry, and motion feasibility.It provides a path to reducing the manual tuning of object-by-object contact rules and evaluates, through a unified benchmark, the practical impact of affordance understanding on robotic manipulation.
The current results are also constrained by the coverage of the affordance model.
For example, contact configurations in some hanging tasks are underrepresented in the model’s training data, and data-collection performance still falls short of that of manual tuning; transparent objects, novel geometries, and ambiguous functional regions also pose challenges. Deformable objects, bimanual manipulation, and in-hand manipulation are not yet included in the current framework. As these capabilities expand, affordance-aware data generation is expected to cover more everyday manipulation scenarios.
References:
https://arxiv.org/abs/2604.11674
TAG384@@
Editor: LRST
Follow ASI in seconds
⭐Like, share, and hit “Looking”—all with one click⭐
Hit the star to stay locked on to Xinzhiyuan’s lightning-fast updates!
