IEEE Robotics and Automation Society IEEE


Online Posts and Relevant Sites:  informal communication and links to resources

Workshops and Meetings: create a sustainable culture of performance evaluation and benchmarking; provide forums for exchanging ideas and approaches.

Major Publications (not counting Proceedings from above workshops): provide references for researchers seeking information on performance evaluation and benchmarking.


These put into action the benchmarks and performance evaluation methods. By organizing and promoting competitions, we put into practice what is discussed during workshops and in publications. Competitions validate the approaches proposed and expose members of the community (especially students) to the concepts. Some of the metrics and test methods are being advanced into draft standards (e.g. the 6DOF localization from the Perception Challenge formed the technical foundation for an ASTM standard being balloted under E57 Committee on 3D Imaging Systems).

  • ICRA'12 Mobile Microrobotics Challenge (MMC) Minneapolis, USA
  • ICRA'12 Solutions in Perception Challenge (SPC) Minneapolis, USA
  • ICRA'12 Virtual Manufacturing Automation Challenge (VMAC) Minneapolis, USA
  • ICRA'11Mobile Microrobotics Challenge (MMC) May 12, 2011 - Shanghai, China
  • ICRA'11Solutions in Perception Challenge (SPC) May 11, 2011 - Shanghai, China
  • ICRA'11 Modular and Reconfigurable Robot Challenge (MRRC) May 10, 2011 - Shanghai, China
  • ICRA'11 Virtual Manufacturing Automation Challenge (VMAC) May 10, 2011 - Shanghai, China

List of outreach activities outside the RAS

The TC has coorganized events with Euron and workshops at RSS. Other initiatives have been carried out in collaboration with the RoboCup organization and in conjunction with the US Department of Homeland Security (DHS). The RoboCup Rescue International competitions rely on quantifiable measures of robot performance within draft and existing test methods for evaluating their ability to assist in disaster response. The best-performing robot teams are invited to participate in the NIST-DHS Response Robot Evaluation Exercises, typically held every 18 months at Disaster City, a major responder training facility in the United States.