Yifei Simon Shao and co-authors received the ICRA 2026 Best Conference Paper Award for SymSkill, a framework designed to help robots learn and combine manipulation skills from a small amount of demonstration data.
Long-horizon manipulation requires a robot to complete tasks made up of multiple dependent steps. A robot putting an object into a cupboard, for example, must understand not only how to move the object, but that the cupboard door must first be opened and may need to be closed afterward.
SymSkill addresses this challenge by learning both physical skills and the symbolic relationships that determine when those skills can be used. In this Q&A, Shao discusses the origins of the project, its relationship to imitation learning and task-and-motion planning, the value of play data, and the system’s current limitations.
Q: Could you begin by telling us about your background and what first attracted you to robotics?
Yifei Simon Shao: I became interested in engineering quite early. In high school, I had a physics teacher who gave me the freedom to explore projects outside the normal curriculum. I built a small firefighting robot with four omnidirectional wheels, so it could move in any direction and point a hose toward a detected fire.
What interested me was that the project required both hardware and software. We had to build the physical system, write the code, work with sensors and think about control. Robotics brings many different fields together, and that was what initially attracted me to it.
As an undergraduate, I continued working on robotics, including autonomous driving and simultaneous localization and mapping. At that stage, much of the work involved systems engineering: taking methods that were already known and making them operate reliably together.
During my master’s degree at the University of Michigan, I worked on safety for reinforcement-learning systems. We studied how to provide safety guarantees even when there were tracking errors. That experience exposed me to a more rigorous style of research.
At the University of Pennsylvania, I have had more freedom to explore research that connects ideas from different areas. SymSkill emerged from looking at imitation learning, task-and-motion planning and reactive policy learning and asking whether their respective strengths could be combined.
Q: What is long-horizon manipulation, and why is it difficult for robots?
Yifei Simon Shao: Long-horizon manipulation refers to tasks that require a robot to perform a sequence of actions over an extended period. The difficulty is not simply executing each individual movement. The robot must also understand the dependencies between those movements.
Consider putting something into a pan. Before the object can be placed inside, the lid may need to be removed. If the robot is only learning a direct mapping from observations to actions, it may learn that sequence from demonstrations. But as the number of steps increases, the dependencies between actions become increasingly difficult to represent.
For a task with one or two steps, a learning system may be able to reproduce the demonstrated behavior directly. For a task involving 10 or 20 steps, it becomes much harder. The robot has to know which conditions matter, which actions must happen first, and which actions are unnecessary in a particular situation. Extending the robot model context alleviates the problem but doesn’t solve it.
SymSkill tries to learn two things simultaneously; that is, how to perform an action and what matters for deciding when that action can be performed.
Q: What limitations in existing approaches were you trying to address?
Yifei Simon Shao: Imitation-learning methods can produce reactive and dexterous behavior, but they often require a large amount of data. A policy may be trained from hundreds of demonstrations of one task, but a different task may require another policy and another large collection of demonstrations.
Task-and-motion planning provides explicit structure. It can represent actions, conditions and dependencies, which makes it possible to compose actions into longer plans. However, these systems can be slow and creating them may require a great deal of manual specification by an expert.
SymSkill is intended to reduce the amount of data required compared with conventional imitation learning, while also reducing the planning time and manual specification associated with task-and-motion planning.
It is not the first system to combine learning from demonstration with planning. Previous work, including work from researchers at MIT, CMU, and Princeton, has learned components for planning from demonstrations. One difference in SymSkill is that we aim for data efficiency at both the high level, where symbolic planning takes place, and the low level, where physical policies execute the actions.
Q: What does “symbol and skill co-invention” mean?
Yifei Simon Shao: The symbols describe what matters in the task. The skills describe how the robot physically achieves it.
A symbol might represent a relationship such as an object being inside a cupboard, a lid being open, or one object being positioned relative to another. A skill is the motion policy used to produce that relationship, such as opening the door or placing the object inside.
The important property we are trying to obtain is compositional generalization. If a robot learns useful skills and the symbolic conditions associated with them, it should be able to reuse and recombine those elements in a new situation.
For example, opening the lid of a pan and opening the lid of a pot are closely related actions. Current physical-learning systems often perform best when the objects used during training and testing are identical or very similar. By separating reusable skills from the symbolic conditions under which they apply, we want to make it easier to compose learned behavior into new task sequences.
Learning everything jointly is not necessarily the only possible route to long-horizon manipulation. But it is one way of making the demonstration data more reusable and of introducing structure that is difficult to obtain from a single monolithic policy.
Q: Why is data efficiency important, and what role does play data have in SymSkill?
Yifei Simon Shao: Data efficiency is partly about the quantity of data, but it is also about how the data is collected.
A conventional demonstration-collection process often begins by resetting a scene to a particular starting state. An operator then demonstrates the same task many times. Those demonstrations are combined to train a policy for that task. If you want the robot to perform 100 different tasks, you may need to collect many demonstrations for each one.
Play data is less structured. Instead of repeatedly demonstrating a predefined task and labeling every action, an operator interacts with the environment more freely. The system can then look for recurring movements, relationships and dependencies within that continuous data.
This could make data collection more passive. Imagine collecting human activity in a household. It would be difficult to ask someone to perform every action 100 times and label each example. A more natural approach would be to observe ordinary activity and identify which actions occur, which objects are involved, and what tends to happen before something else.
Every environment also has its own dependencies and habits. The order in which tasks are performed may differ from one household to another. Play data offers a possible way for a robot to learn those local patterns without requiring every task to be specified in advance.
Q: Can you explain how the “store cheese” example works?
Yifei Simon Shao: The task involves opening a cupboard door, putting the cheese inside, and closing the door.
The system first needs estimates of the objects and their poses. It examines the trajectory to identify which object is moving and to estimate the beginning and end of each movement.
A vision-language model is then used to identify the relevant reference object. If the cupboard door is being moved, for example, the system determines that the door’s movement should be understood relative to the cupboard rather than relative to the cheese.
This produces data describing which object moved, what it moved relative to, and the relative pose achieved at the end of the movement. From this, the system can identify useful predicates, such as “the cheese is inside the cupboard” or “the door is open.”
Across the demonstrations, it can then discover dependencies. Whenever the cheese is placed inside the cupboard, the cupboard door is already open. The system can therefore infer that an open door is a precondition for putting the cheese inside.
The logic is simple when stated explicitly. The challenge is enabling the robot to extract that structure from demonstrations and connect it to the physical policies required to perform each step.
Q: Does the system’s data efficiency also reduce the amount of computation required?
Yifei Simon Shao: The learned components of our system are relatively lightweight and can run on a CPU. That is very different from large models containing billions of parameters.
However, the comparison needs to be made carefully. SymSkill receives structured information about object poses and trajectories. Producing that information requires a separate perception system, and that perception system may itself be computationally demanding.
In some large learning-based approaches, perception, task understanding, planning and policy execution are all contained within one model. In SymSkill, those elements are more explicitly separated.
I would not claim that SymSkill necessarily solves the overall computation problem. The narrower point is that, once reliable structured perception is available, the symbolic and policy-learning components can be lightweight. The system also makes some of its internal structure visible, rather than placing all aspects of the task inside a single black-box model.
Q: What are the principal limitations of the current system?
Yifei Simon Shao: One limitation is that the experiments involve rigid objects whose poses can be tracked. We also use object velocity as a way of dividing trajectories into different skills. Velocity is a useful proxy for identifying when one action ends and another begins, but it is not itself the abstraction we ultimately want the system to discover.
Another limitation is that the symbolic representation is based primarily on relative poses. Relationships such as one object being inside, beside or on top of another provide a useful approximation of what matters in many tasks, but this is an aggressive simplification of the world.
Pose tracking is also still difficult. Our current approach assumes that the relevant objects can be identified and tracked throughout the demonstration. Reducing that dependence would make the method more practical.
These assumptions helped us study the central question of jointly learning skills and symbolic structure, but they also define the areas in which the approach needs to improve.
Q: What are the next steps for this research?
Yifei Simon Shao: An important next step is to discover abstractions more directly from the data, rather than relying on predefined cues such as object velocity and relative pose.
Ideally, the system would automatically discover a hierarchy. At the highest level, the robot has an overall task. At an intermediate level, it identifies the subgoals required to complete that task. At the lowest level, it executes the motor actions needed to achieve each subgoal.
The number and structure of those levels should not have to be decided in advance. A person might describe opening a lid as a single action or as two actions, while the algorithm may discover that five meaningful stages are required.
There is related research in model-based reinforcement learning and world models that tries to extract reusable actions and changes in observation from demonstration or play data. However, automatically learning a complete hierarchy remains underexplored.
In everyday terms, the long-term goal is to create robots that can learn how tasks are organized in a particular environment, understand the habits of the people around them, and carry out useful sequences of actions reliably.
Emmet Cole, Science Communicator
