Universidad Carlos III de Madrid, Spain
Angel P. del Pobil
Jaume I University, Castellon, Spain
National Institute of Standards and Technology, USA
Workshops: create a sustainable culture of performance evaluation and benchmarking; provide forums for exchanging ideas and approaches.
- Proposals for experimental protocols for Robotics Research, RSS Workshop , June 27, 2013, Berlin, Germany
- Metrics of sensory motor integration in robots and animals, Workshop, October 12, 2012, Vilamoura, Algarve, Portugal, Half-day IROS2012 workshop
- Safety, Security, and Rescue Robotics Summer School, September 10-14, 2012 Alanya, Turkey. Students and early career researchers experience test method development process first-hand.
- From theory to practice of performance comparison and result replications in Robotics Research, Workshop, Australia - July 12, 2012, Sydney, NSW, Australia RSS 2012 workshop
- Conditions for Replicable Experiments and Performance Comparison in Robotics Research, Workshop, Minneapolis MN USA, Full-day ICRA 2012 workshop.
- Performance Metrics for Intelligent Systems (PerMIS'12) March 20-22, 2012, College Park, MD Three-day Workshop
- Towards Replicable Experiments in Robotics Research, San Francisco, CA USA, ICRA 2011 Full-day Workshop
- Performance Evaluation and Benchmarking for Intelligent Robots and Systems with Cognitive and Autonomy Capabilities , Taiwan, IROS 2010 Half-day Workshop
- Performance Metrics for Intelligent Systems (PerMIS'10) Three-day Workshop September 30, 2010 - Baltimore, MD
- Benchmarking Intelligent (Multi-)Robot Systems August 20, 2010 - Lisbon, Portugal, Full-Day Workshop
- Good experimental methodology in robotics and replicable robotics research June 28, 2010 - Zaragoza, Spain, RSS 2010 Half-day Workshop.
- The Role of Experiments in Robotics Research Anchorage, Alaska, USA, ICRA 2010 Full-day Workshop.
- Performance Evaluation and Benchmarking for Next Intelligent Robots and Systems St. Louis, MO, USA Full-Day IROS'10 Workshop
- Performance Metrics for Intelligent Systems (PerMIS'09) Workshop, September 21, 2009 – USA Three-day Workshop
- Good Experimental Methodology in Robotics Workshop June 28, 2009 - Seattle, Washington, USA, RSS 2009 Full-day Workshop
Major Publications (not counting Proceedings from above workshops): provide references for researchers seeking information on performance evaluation and benchmarking.
- International Journal of Intelligent Control and Systems (IJICS) Special Issue "Quantifying Performance of Intelligent Systems," Madhavan, R., Tunstel, E., Messina, E., eds., Volume 16, Number 2, June 2011.
- Quantifying Performance of Intelligent Systems," International Journal of Intelligent Control and Systems Special Issue; Volume 16, Number 2, June 2011
- Performance Evaluation and Benchmarking of Intelligent Systems Edited Book Volume, Raj Madhavan, Edward Tunstel and Elena Messina (eds.), Springer, ISBN: 978-1-4419-0491-1, September 2009.
- Autonomous Robots Journal, Special Issue (R. Madhavan, C. Scrapper, and A. Kleiner (Eds.)) November 1, 2009 - New York
- Performance Evaluation and Benchmarking of Robotic and Automation Systems [TC Spotlight] Madhavan, R.; del Pobil, A.P.; Messina, E.; Robotics & Automation Magazine, IEEE, Volume: 17 , Issue: 1; Publication Year: 2010 , Page(s): 120 – 122.
These put into action the benchmarks and performance evaluation methods. By organizing and promoting competitions, we put into practice what is discussed during workshops and in publications. Competitions validate the approaches proposed and expose members of the community (especially students) to the concepts. Some of the metrics and test methods are being advanced into draft standards (e.g. the 6DOF localization from the Perception Challenge formed the technical foundation for an ASTM standard being balloted under E57 Committee on 3D Imaging Systems).
- ICRA'12 Mobile Microrobotics Challenge (MMC) Minneapolis, USA
- ICRA'12 Solutions in Perception Challenge (SPC) Minneapolis, USA
- ICRA'12 Virtual Manufacturing Automation Challenge (VMAC) Minneapolis, USA
- ICRA'11Mobile Microrobotics Challenge (MMC) May 12, 2011 - Shanghai, China
- ICRA'11Solutions in Perception Challenge (SPC) May 11, 2011 - Shanghai, China
- ICRA'11 Modular and Reconfigurable Robot Challenge (MRRC) May 10, 2011 - Shanghai, China
- ICRA'11 Virtual Manufacturing Automation Challenge (VMAC) May 10, 2011 - Shanghai, China
List of outreach activities outside the RAS
The TC has coorganized events with Euron and workshops at RSS. Other initiatives have been carried out in collaboration with the RoboCup organization and in conjunction with the US Department of Homeland Security (DHS). The RoboCup Rescue International competitions rely on quantifiable measures of robot performance within draft and existing test methods for evaluating their ability to assist in disaster response. The best-performing robot teams are invited to participate in the NIST-DHS Response Robot Evaluation Exercises, typically held every 18 months at Disaster City, a major responder training facility in the United States.
Performance Evaluation & Benchmarking of Robotic and Automation Systems
Welcome to the official website of the IEEE RAS Technical Committee on Performance Evaluation and Benchmarking of autonomous systems
TC-PEBRAS serves as a forum to address performance evaluation and benchmarking for robotic and automation systems, in general.
The TC-PEBRAS contributes to the progress of performance evaluation and benchmarking, focusing in intelligent robots and systems, including those with some high level cognitive capabilities and some degree of autonomy, by providing a forum for researchers and engineers in the industry to exchange their on-going work and ideas in this regard. A purpose of the TC is IN PARTICULAR to achieve better and agreed ideas on how to define and measure system level characteristics like 'autonomy', 'cognition' and 'intelligence'.