Learning from Demonstration

Learning from Demonstration

Prerequisites

Linear Algebra and multivariable calculus
Classical Dynamics
Robotics (Optional, for Implementation)
- Motion Planning and Optimal Control
- Basic Machine Learning and Reinforcement Learning

Motivation

Enabling robots to learn to autonomously perform new tasks from scratch (i.e., without any prior knowledge) can be extremely difficult, particularly when trying to encode complex behaviors into generalized controllers that can generate successful performance even from new states. To solve this, Learning from Demonstration (LfD) enables robots to learn how to autonomously perform a task by observing and imitating a set of successful, expert demonstrations.

Because these methods do not require users to manually construct, encode and program task objectives and resulting motion planners and controllers, these methods allow novice users who do not necessarily have background in programming or robotics to successfully demonstrate and “teach” a robot how to perform novel tasks by simply providing task demonstrations. These methods are particularly useful for tasks that are difficult to formally define or engineer by experts. Furthermore, when these methods are successfully implemented into a robotic system, they provide a easy way to program new tasks or behaviors to the robot without expert (robotic) knowledge.

Learning from demonstration encompasses a wide range of research topics and methods, including Learning from Demonstration (LfD), imitation learning, programming by demonstrations (PbD), and inverse reinforcement learning (IRL) methods. A key goal of all LfD methods is to learn some form of generalized encoding (e.g., a controller, policy) of how to successfully perform a task. Different methods learn different encodings to generalize performance, as will be discussed later, but fundamentally LfD methods do not simply replay a demonstration, but generalize from the set of demonstrations how to perform the task.

Before delving into details on the different types of methods, we will first cover some basics that applies to all methods.

Course Content

Foundations of Learning from Demonstration

LfD methods are primarily data-driven methods that seek to how to perform a task from observations of successful demonstrations. They can do so either by directly learning a motion plan or policy, or inferring the task objective function which is then used to generate an optimal policy.

Basic LfD Formulation

Nomenclature and Notation

Depending on whether the LFD problem is formulated from a optimal control (Pontryagin) or reinforcement learning (Bellman) perspective, the notation will differ, though they are analagous. Both notations are provided below:

Notation	Optimal Control (Pontryagin)	Reinforcement Learning (Bellman)
State at time $t$	$$x_t$$	$$s_t$$
Action at time $t$	$$u_t$$	$$a_t$$
Task Objective Function	Cost Function: $J(x_t,u_t)$	Reward Function: $r(s_t,a_t)$
Dynamics	$x_{t+1} = f(x_t,u_t)$ or $p(x_{t+1} \| x_t, u_t)$	$$p(s_{t+1} \| s_t,a_t)$$
Observations	n/a	$o_t$
Demonstrations	$D = [(x^n_t, u^n_t)]$ for $t \in [0,T]$ for $N$ demonstrations	$D = [(s^n_t, a^n_t)]$ for $t \in [0,T]$ for $N$ demonstrations
Optimal Action (to be learned)	Optimal Control: $u^*(x)$	Learned Policy: $\pi_\theta(a\|s)$ or learn reward parameters $r_\psi(s,a)$ and use to learn optimal policy $\pi^*(a\|s)$

The general problem formulation for LfD is as follows:

Given the expert demonstration data set $D$, consisting of a list of state-action pairs ($(s,a)$ or $(x,u)$)
Solve for the optimal task policy $\pi^*(s)$ that matches the expert demonstrations. This can be done either by:
- Solving for the reward function $R(s,a)$, then using that to generate the optimal policy $\pi(s)$
- or directly solving for $\pi(s)$ that best matches the expert demonstrations.

All methods involve some form of learning through a minimization problem that minimizes the distance between the reference expert demonstration actions and the output of the learned policy. Defining the learned parameters representing the task and resulting policy depends on the type of method used and how it represents the task and policy. For now, we will just call them $\theta_R$. Then, we can define the problem as:

$ \underset{\theta_R}{\text{minimize}} \qquad \text{distance}(a_{d}(s_{d})-\pi(s))$,

where $(a_d, s_d)$ reflect the expert state-action pairs and some distance measure (e.g. L2-norm, NN loss functions) are used to match the expert and learned policy outputs.

Learning from demonstration methods use two main steps:

Gathering Demonstration dataset $D$
Learning from the data to generate a task policy $\pi(s)$

If and how the method specifies the two phases of the data collection determines the following properties of the method.

Data-Gathering Methods

Because demonstration learning methods are fundamentally data-driven learning approaches, one of the main topics when designing LfD paradigms is data collection. Demonstration Data collection methods fall into three main categories, each with their own pros and cons:

Kinesthetic Teaching: user physically moves the robot
Teleoperation: user controls the robot through interface
Observational learning: robot learns from observations of demonstration

Kinesthetic Teaching

Kinesthetic Teaching of Robot Demonstrations for Playing Minigolf. User physically moves the robot through successful task execution to generate a demonstration (Image Source: Khansari-Zadeh, et al. Learning to play minigolf: A dynamical system-based approach. Advanced Robotics (2012). DOI )

The first category of data collection is kinesthetic teaching methods, where the user physically moves robot to generate demonstrations. Because these demonstrations directly record the motions of the robot using its onboard sensors (e.g., joint states and joint torques), a major benefit of kinesthetic teaching is no mapping or correspondence is needed between the demonstration data and the robot motion control itself.

However, one of main drawbacks of kinesthetic teaching are that it can be cumbersome for the user to physically move the robot in the desired motion. Because the user needs two arms to move one robot arm, it can be challenging for the user to do so in a fluid motion and demonstrating bimanual tasks can be challenging. Dextrous manipulation tasks that require finger control can also be difficult to move and demonstrate in kinesthetic settings. Furthermore, it can be difficult or unintuitive for novice users to generate demonstrations, particularly for high degree of freedom (DOF) or complex dynamical systems. These dynamical systems can be difficult for users to relate to the kinematics and dynamics of the human motion, making generating desired robot motion difficult.

Teleoperation

In teleoperation, the user controls the robotic system through an interface. Similar to kinesthetic teaching, the demonstrations directly record the robot states through its onboard sensors (e.g., joint angles and torques); however, the experimental design requires developing an interface from user inputs to robot motion and control. The ease of use of the robotic systems, and the resulting quality of the demonstration dataset, can be directly affected both by the user’s experience with the teleoperated robotic system and the design of the interface.

These systems allow the demonstrator to control the robotic system from a distance or even remote locations, allowing users to not be physically present with the robot to provide demonstrations. In addition, these methods facilitate safe control for users when controlling heavy machinery that could be dangerous to users in close proximity, as well as enabling control for tasks where the robotic system needs to move over distances, such as when teaching motion patterns or navigation.

Teleoperation interfaces used to control robots can range from simple joysticks (Left: Universal Robots Wiretank Joystick) and tablet interfaces to complex interfaces, including surgical robots (Middle: Da Vinci Surgical Robot System) and exoskeletons (Right: Capio Dual-arm Exoskeleton ).

Common interfaces for teleoperation include joysticks, graphical interfaces, haptic interfaces, or more complex multimodal interfaces. Common examples of teleoperation interfaces include:

Joysticks are typically used to directly control the motion of the system, either through direct joint control or through end-effector motion control in conjunction with an inverse dynamics controller.
Graphical interfaces, such as tablet interfaces, can be used to control a robot by either mimicking the motion on the tablet surface (e.g., such as ``drawing’’ the desired motion in space for the system to follow) or by communicating high-level information (abstract or symbolic information) to the robotic system.
Haptic interfaces, where a teleoperation interface (such as a joystick is combined with haptic force-feedback sensors) are considered bilateral interfaces that both send motion information to the robotic system and provide haptic force-feedback to the user. These are particularly useful in tasks where force-feedback is critical, such as in teloperated surgical robots.
Finally, teleoperation interfaces can be more complex devices, combining complex dynamical motion interfaces, such as exoskeletons, with multimodal sensor systems (vision, haptic, etc.) to give realistic, real-time feedback to the user.

Shadowing is a subset of teleoperation demonstration gathering, where the robot mimics or shadows the demonstrator’s motions, typically through some type of motion capture interface, and the robot’s motions are recorded as demonstration data. With these interfaces, the demonstrator does not need familiarity with interacting with or controlling the robotic system. Instead, the human demonstrator independently performs the task and the robot follows the motions. This requires the mapping between the motion capture sensors on the human and the robot joint motions to be defined such that the robot can move analogously to the human user, such as with a humanoid robot or haptic wearable glove.

Shadowing interfaces, including the Left: Capio Dual-arm Exoskeleton and Right: WEART TouchDIVER G1 haptic glove shown here, require some form of mapping from the demonstration sensors to the robotic system during recording, so that the robotic system shadows the demonstrator’s motions and the robot motion is directly recorded as demonstrations.

Observational Learning

Observational learning interfaces can include vision, wearable, and other motion capture devices. All require some form of embodiment mapping from the demonstration sensor data to the robotic system. Top: DexPilot : Vision-based Learning using only Visual Camera Feedback. Bottom: DexCap Hand Motion Capture for Manipulation. (Bottom Left) A combination of hand pose data from the haptic gloves and visual feedback from the camera on the chest is used to collect demonstration data. (Bottom Right) Embodiment Mapping from human hand data from glove to 4-finger robotic hand.

With observational learning methods, the robot learns by observing the task being done successfully and learns from “watching” these demonstrations. The observations are typically of human demonstrations, but may also be from another robotic system successfully performing the task. The key characteristic of these methods is that the learning is based on observations of these demonstrations, without any physical guidance of the robot’s motions themselves. As such, acquiring demonstrations can be easy for the demonstrator, as no familiarity with the robotic system may be necessary. Furthermore, since these demonstrations need not be acquired with the robotic system, demonstrations can be acquired from remotely, making it easy to acquire larger numbers of demonstrations.

The demonstrations can be recorded through a number of motion tracking mediums— vision, wearable devices, exoskeletons, standard motion tracking systems (e.g., Optitrack, Microsoft Kinect). However, these methods require a good mapping between the observational data and the kinematics of the robotic system. If the kinematics between the observations and system are similar, for example between human demonstrations collected using a motion tracking system and a humanoid robot, the mapping can be more easily solved. However, if the dynamics are significantly different, the mapping can be hard to solve for— this is known as the correspondence problem, discussed further below.

Data Collection Methods – Dropdown Table

Select the correct option in each cell, then click Check Answers.

Data Collection Type	Demonstration Data Mapping Required	Familiarity with Robot System Needed	Remote Collection Possible
Kinesthetic Teaching
Teleoperation
Shadowing
Observational Learning

_{This table was based on Table 1 from:Ravichandar, Harish, et al. "Recent advances in robot learning from demonstration." Annual review of control, robotics, and autonomous systems 3.1 (2020): 297-330. ![doi](https://doi.org/10.1146/annurev-control-100819-063206)}

Conceptual Exercise

Drag each type of demonstration interface example to the correct LfD Data Collection category: