The OmniRetarget research project achieved a double at ICRA 2026, winning both the Best Conference Paper Award and the Best Paper Award on Robot Manipulation and Locomotion.
OmniRetarget proposes a way of converting human demonstrations into feasible motions for humanoid robots. Rather than asking a robot to copy a human pose exactly, OmniRetarget preserves the important relationships between the body, objects and surrounding terrain.
We spoke with lead author Lujie Yang, a member of Technical Staff at Amazon’s FAR (Frontier AI & Robotics), about interaction-preserving motion retargeting, generating training data and the connection between data quality and robot learning.
Q: Congratulations to you and your team on winning the award. Before we get into OmniRetarget, could you tell readers a little about your background and how you came to work on this project?
Lujie Yang: I am currently a member of technical staff at Amazon Frontier AI & Robotics. I completed my undergraduate studies at UC Berkeley, where I worked with Professor Francesco Borrelli on control theory for autonomous driving. I was initially attracted by the mathematics and how rigorous control theory could drive vehicles autonomously.
I then completed my Ph.D. at MIT with Professor Russ Tedrake, combining control theory, optimization and machine learning for dexterous, contact-rich manipulation. What excites me most is seeing rigorous mathematics deployed on real robots to make them perform interesting tasks.
In part, OmniRetarget grew from an earlier project called PhysicsGen, in which we transferred demonstrations involving human fingers manipulating a small cube onto much larger robot arms manipulating a box. We wondered whether similar techniques could be applied to whole-body loco-manipulation. When I explored existing methods, I found that retargeting itself was a major bottleneck, so I decided to develop my own system.
Q: For readers new to these topics, what is motion retargeting, and why is it important?
Lujie Yang: Motion retargeting converts motion performed by a source agent, typically a human, into physically feasible motion for a target agent, such as a humanoid robot.
Collecting demonstrations directly on robots through teleoperation or manual programming requires considerable time and resources and can wear the hardware. Human demonstrations are much easier to collect and are already available through motion capture, video and other sources.
If we can reliably translate those abundant human motions into high-quality robot motions, we can generate much more training data and teach humanoids skills that humans already know how to perform.
Q: Why do conventional retargeting methods struggle when transferring human motion onto a robot?
Lujie Yang: Existing methods commonly use keypoint matching, which tries to match important global body positions between the human and robot. But humans and robots can have very different heights and proportions.
The Unitree G1 used in our experiments is approximately 1.3 meters tall, while a human demonstrator might be 1.8 meters tall. If you simply scale the human keypoints down, you can produce physically implausible results. When a human holds a box, for example, direct scaling may position the robot’s hands inside the box rather than on its surface.
Existing methods also frequently neglect interactions with objects and the environment. If a robot is carrying a box or climbing a platform, maintaining the correct contacts is central to the task. Matching body positions is not enough if its hands lose contact with the box or its feet slide across the platform.
Q: OmniRetarget preserves interactions rather than copying the human’s exact pose. How does that work?
Lujie Yang: It’s about preserving the relationships between the body and its surroundings.
OmniRetarget represents spatial and contact relationships using an interaction mesh. The mesh connects the body to nearby objects and terrain and encodes information such as relative distances, geometry and contact. During optimization, the system adapts the motion to the robot’s body while minimizing changes to that interaction structure.
Many tasks are defined by interactions rather than poses. When carrying a box, what matters is that both hands are correctly positioned on the box—not the precise global coordinates of the human demonstrator’s hands. Similarly, when climbing, stable contact with the environment is more important than exactly reproducing the demonstrator’s posture.
This also changes the broader learning process. Existing approaches may start with lower-quality reference motions and try to compensate with complex reward engineering. We try to solve the problem at its source by producing better reference motions. With higher-quality data, we can use a much simpler reinforcement-learning formulation.
Q: One demonstration can be adapted to different objects, terrains and robot embodiments. How important is this one-to-many capability?
Lujie Yang: It is one of the most exciting aspects of OmniRetarget because collecting high-quality demonstrations is often a major bottleneck in robot learning.
Traditionally, even a small change—such as making a table five centimeters taller or moving an object—may require another round of data collection.
It is like teaching a new employee to pick up a box and having them believe the box must always be the same size and in exactly the same place. If the box becomes smaller or moves to the left, you have to teach them again.
With OmniRetarget, one demonstration can generate an entire family of related demonstrations involving different object sizes and positions, terrain heights and robot bodies. Instead of collecting thousands of demonstrations containing minor variations, we can collect a smaller number and systematically expand them into a richer dataset.
Q: The project demonstrates box carrying, platform climbing, crawling, rolling and a wall flip. Who doesn’t love a wall flip? But in your view, which result best captures what OmniRetarget can do?
Lujie Yang: The wall flip is probably the most visually striking demonstration. It shows the robot’s dynamic capabilities and the importance of correctly preserving the interaction between its foot and the wall.
However, the box-carrying experiments better capture the one-to-many capability. You demonstrate how to pick up an object once and then generate variations. In our hardware experiments, the robot could bend over to pick up objects positioned to its left or right, or rotate its hips while carrying the box.
The idea is that you teach the interaction once and the system adapts it to different positions, sizes and configurations.
Q: What enabled the policies to transfer directly from simulation to the physical robot?
Lujie Yang: The quality of the retargeted motions is central.
Previous methods may produce penetration, where the robot intersects with the ground, itself, or an object. They may also produce foot-skating, where a foot that should remain planted slides across the surface.
These are not good behaviours for a reinforcement-learning algorithm to track and are especially difficult to reproduce on a physical robot. OmniRetarget minimizes these artifacts while preserving important contacts and spatial relationships.
Because the reference motions are physically plausible, the policy does not have to compensate for as many errors. That allowed us to achieve zero-shot sim-to-real transfer using only proprioceptive observations and a relatively minimal training formulation.
Q: What are OmniRetarget’s current limitations? And what is next for this line of research?
Lujie Yang: OmniRetarget currently assumes that the original reference motion is accurate. If the source contains an incorrect interaction—for example, if the human hand is reconstructed on the wrong side of an object—the system cannot yet recognize and correct that error automatically.
We want to scale up data generation using sources including motion capture, video and teleoperation. Those larger quantities of high-quality data could then support different learning algorithms. The longer-term goal is to contribute to robot foundation models that can perform many tasks and generalize across scenes and environments.
OmniRetarget primarily addresses physical intelligence: the robot’s ability to move and interact reliably with the world. At the same time, vision-language models and other systems are advancing the cognitive intelligence of the robot brain. Combining physical and cognitive intelligence could produce robots that can both reason about tasks and interact robustly with the real world.
Q: Finally, how would you explain the significance of OmniRetarget to somebody who knows nothing about robotics?
Lujie Yang: Imagine teaching a robot to help with household chores by showing it how to pick up a laundry basket.
You do not want the robot to copy the position of every joint in your body. What matters is the interaction: where it should stand in relation to the basket and where it should place its hands or arms.
That is essentially what OmniRetarget does. Rather than asking robots to imitate humans exactly, we teach them to preserve the important interactions with the world. This allows robots with very different bodies to learn useful skills from human demonstrations.
The approach is not necessarily limited to humanoids. In principle, it could also be applied to dual-arm robots or even quadrupeds, bringing us closer to robots that can learn from people and assist them in homes, workplaces and other environments.
Emmet Cole, Science Communicator
