Objectives

Benchmarking service robots has drawn increasing interest from researchers and professionals in recent years. A common form of benchmarking service robots is service robot competitions, such as RoboCup@Home and RoCKIn.

    Since 1997, RoboCup has annually attracted in the thousands of participants from dozens of countries to demonstrate their latest progress on robotic research and developments in well-structured competitions. One of the greatest successes of RoboCup is providing common benchmark platforms or problems (e.g., soccer, rescue, domestic service, manufacturing, and logistics) for testing and evaluating robot technologies. Evaluating performance in task-based robot competitions, including RoboCup, is challenging, as the evaluation is about the overall task, such as number of scored goals. Although other competitions, such as RoboCup@Home, decompose the task into parts to be quantitatively evaluated, they still include subjective judging by humans.

    Specific quantitative measurements on robot performance would enable to directly and accurately identify the strengths and weaknesses of robot technologies through tests or competitions. We are therefore organizing a competition and a workshop on benchmarking service robots towards further understanding and proposing quantitative measurements in robotics research. Our workshop/competition is a sub-event of RoboCup 2015 with one day for presentations and discussions and three days for the demo competition.

 

Back