IEEE Robotics and Automation Society IEEE



fabio bonsignorio
Fabio Bonsignorio
Universidad Carlos III de Madrid, Spain
This email address is being protected from spambots. You need JavaScript enabled to view it.

Angel P. del Pobil
Jaume I University, Castellon, Spain
This email address is being protected from spambots. You need JavaScript enabled to view it.

Signe Redfield
Signe Redfield
Naval Research Laboratory, USA
This email address is being protected from spambots. You need JavaScript enabled to view it.

Founding Date:

October 2009

Member Count:

About 100


Online Posts and Relevant Sites:  informal communication and links to resources

Workshops and Meetings: create a sustainable culture of performance evaluation and benchmarking; provide forums for exchanging ideas and approaches.

Major Publications (not counting Proceedings from above workshops): provide references for researchers seeking information on performance evaluation and benchmarking.


These put into action the benchmarks and performance evaluation methods. By organizing and promoting competitions, we put into practice what is discussed during workshops and in publications. Competitions validate the approaches proposed and expose members of the community (especially students) to the concepts. Some of the metrics and test methods are being advanced into draft standards (e.g. the 6DOF localization from the Perception Challenge formed the technical foundation for an ASTM standard being balloted under E57 Committee on 3D Imaging Systems).

  • ICRA'12 Mobile Microrobotics Challenge (MMC) Minneapolis, USA
  • ICRA'12 Solutions in Perception Challenge (SPC) Minneapolis, USA
  • ICRA'12 Virtual Manufacturing Automation Challenge (VMAC) Minneapolis, USA
  • ICRA'11Mobile Microrobotics Challenge (MMC) May 12, 2011 - Shanghai, China
  • ICRA'11Solutions in Perception Challenge (SPC) May 11, 2011 - Shanghai, China
  • ICRA'11 Modular and Reconfigurable Robot Challenge (MRRC) May 10, 2011 - Shanghai, China
  • ICRA'11 Virtual Manufacturing Automation Challenge (VMAC) May 10, 2011 - Shanghai, China

List of outreach activities outside the RAS

The TC has coorganized events with Euron and workshops at RSS. Other initiatives have been carried out in collaboration with the RoboCup organization and in conjunction with the US Department of Homeland Security (DHS). The RoboCup Rescue International competitions rely on quantifiable measures of robot performance within draft and existing test methods for evaluating their ability to assist in disaster response. The best-performing robot teams are invited to participate in the NIST-DHS Response Robot Evaluation Exercises, typically held every 18 months at Disaster City, a major responder training facility in the United States.

Performance Evaluation & Benchmarking of Robotic and Automation Systems

Welcome to the official website of the IEEE RAS Technical Committee on Performance Evaluation and Benchmarking of autonomous systems

TC-PEBRAS serves as a forum to address performance evaluation and benchmarking for robotic and automation systems, in general.

The TC-PEBRAS contributes to the progress of performance evaluation and benchmarking, focusing in intelligent robots and systems, including those with some high level cognitive capabilities and some degree of autonomy, by providing a forum for researchers and engineers in the industry to exchange their on-going work and ideas in this regard. A purpose of the TC is IN PARTICULAR to achieve better and agreed ideas on how to define and measure system level characteristics like 'autonomy', 'cognition' and 'intelligence'.