Replicable robotics research and benchmarking

Organiser(s): Fabio BONSIGNORIO, The BioRobotics Institute, SSSA and Heron Robots, Italy (,;
Jan VENEMAN, Tecnalia, Spain (
Content: The still ongoing internet revolution, including the 'internet of things', is mainly focused on information management. Robotics Science and Technologies enable the next step: the possibility of physical interaction with people and the environment in which they live. The existence of objective operational procedures allowing to assess progress, 'step changes' or 'TRL' improvements (or even the bare 'state of the art'), is a critical untapped prerequisite for the good functioning of the MAR roadmapped research process. Those operational assessing procedures (comparable to medicine 'clinical trials' of new drugs) will require extensive testing in realistic environments. Shared infrastructures such as those developed in Echord and Echord++ projects are necessary to contain costs and share best practices and insights. In particular Robotics High Tech startups cannot afford the multimillion investment of proprietary testing infrastructures.

The workshop will present and discuss methodologies for benchmarking and will discuss current projects and infrastructures aimed at their deployment in the market.

As an example of benchmarking technical challenges, there will be an introduction by specialists active in the working group "Benchmarking of Bipedal Locomotion" set up by European research projects on Humanoids and Exoskeletons, a.o H2R, Koroibot, Walkman, Biomot and Balance, followed by a round table on how to build relevant benchmarks in this area.
Agenda of the workshop: 10:45 - 10:55 How to assess robotics research and products? The case of Robot Companions for Assisted Living - Fabio Bonsignorio
10:55 - 11:05 An early model of robotics infrastructure: the ECHORD++ Research and Innovation Facility (RIF) - Filippo Cavallo
11:05 - 11: 15 Technical Issues in Benchmarking: The example of benchmarking of bipedal locomotion in robotics by selected presenters
11:15 - 11:40 Open Round Table Discussion about introduced benchmarking approach with selected invited participants and the audience
11:40 - 12:05 Open Round Table Discussion "Towards a research and innovation infrastructure for robotics" (moderated by Paolo Dario): Comau representative, RIF experiment representative, Cecile Huet, Alois Knoll, Chris Melhuish, Gurvinder Virk
12:05 - 12:15 Conclusions by the organizers for the Roadmapping Process under SPARC/Horizon 2020
  • Paolo DARIO, coordinator of the Topic Group on Robot Companions for Assisted Living, The BioRobotics Institute, Scuola Superiore Sant'Anna, Italy
  • Fabio BONSIGNORIO, coordinator of the Topic Group on Benchmarking and Competitions, The BioRobotics Institute, Scuola Superiore Sant'Anna, Italy and Heron Robots, Italy
  • Alois KNOLL, coordinator of the EU project ECHORD++, TUM, Germany
  • Chris MELHUISH, partner in the EU project ECHORD++ and responsible of the Bristol RIF, Bristol Robotics Laboratory, UK; Comau representative, Italy; ECHORD ++ RIF experiment representative
  • Cecile HUET, Deputy Head of Unit Robotics - A2, European Commission; Gurvinder VIRK, coordinator of the Topic Group on Standardization, Sweden
  • Diego TORRICELLI, H2R FP7 project, coordinator bipedal benchmarking working group, CSIC, Spain
  • Katja MOMBAUR, Koroibot FP7 project Coordinator, University of Heidelberg Germany
  • Nikos TSARAGAKIS, Walkman FP7 project coordinator,, IIT, Italy; Jose Luis PONS, Biomot FP7 project coordinator, CSIC, Spain
  • Anne BAJART, EC, Robotics Research Programme Officer
  • Jan VENEMAN, Balance FP7 project coordinator, expert in ISO and IEC on medical and personal care robots, Tecnalia, Spain