RoboCup-Rescue Robotics and Infrastructure Evaluation Methods

The aim of RoboCup-Rescue Robotics and Infrastructure Evaluation is to provide standarlized evaluation methods that can cope with various research and development needs, and to promote research through competitive evaluation sessions. Since the scope of the robotics and infrastructure project is extremely broad, a single evaluation scheme would not cover entire research needs. Thus, we will provide a series of evaluation benchmark. Obviously, it is not a complete set of benchmark. Evaluation benchmark will be added to cope with diverse and increasing needs. At this moment, we provide a benchamrk evaluation scheme for the most obvious tasks.

Search and Rescue Robot System Evaluation

Search and rescue robotic system evaluation is sub-categoried into three tasks:

Victim Search Task
Victim Rescue Task
Search and Rescue Task

Each task can be situated in either in-door test field or out-door test field. Although evaluation will be made on a single disaster site at beginning, multiple disaster sites will be created to evaluate resource allocation capability of the whole system. General set up of the field and simulated victims will be arranged to maintain reasonable compatibility with AAAI Search and Rescue Competition

Victim Search Task

Objective

The victim search task aims at evaluation of the system's capability of finding victims under the debris. Any set of equipments (robot, remote controlled devices, etc) are acceptable. Special categories dedicated for specific aspects of technologies may be created, such as a full autonomous system category.

Field Set Up

A simulated desaster site is created with debris as close as possible within logistical and other practical constraints, as well as to meet evaluation purpose. The size of disaster site for the in-door evaluation may be 5m by 5m. Numbers of debris are randomly set and the simulated victim is located randomly. Victims are simulated by specially designed dolls or objects which may emit sound, body heat, CO2, or other possible cue that actual victims may generate.

Evaluation

Performance of each system is evaluated on several criteria:

Weighted time average for all victiums found. This corresponds to expected number of victims saved over all victims actually under the debris.
Recover and Accuracy Rate
- Recover Rate: How many victims does the system identified out of all victims under the debris. If the system found 6 victims out of 10 victims actually under the debris, the system has 60% recovery rate.
- Accuracy: How many identification were correct out of all identifications made by the system. If the system spotted 8 places as victims, and actually only 4 of them are correct, accuracy is 50%.
Operational Loading. Numbers of operations that has to made by each human operator. This measure evaluate how many equipments can be operated at the time of disaster where human resource is bottleneck.

Overall evaluation function will be provided shortly to linealize the evaluation criteira above.

Victim Rescue Task

The victim rescue task evaluates how quickly the system can pull out victims under the debris without hurting the victim itself. The location of the victim is assumed to be identified, and this benchmark evalute mostly on capability of the robot to safely remove debris and move victims out from the debris, to finally bring him/her to the first aid station.

Field Set Up

Compatible with search task. The in-door field is set up within 5 m by 5m field with debris and simujlated victims. Simulated victims are dolls with special pressure sensors that measures possible damanges imposed on their bodyt during the rescue task.

Evaluation

Total time to took to rescue all victims
Total damage imposed on victims
Expected number of life saved
Operation Load. Total number of operations made by each human operator during the rescue operation.

Search and Rescue Task

The goal os the search and rescue task is to provide a comprehensive task for search and rescue operation. Basically this is a comobiunation of two tasks mentioned above.

Field Set Up

Evaluation

Performance of each system is evaluated on several criteria:

Weighted time average for all victiums found and rescued. This corresponds to expected number of victims saved over all victims actually under the debris.
Coverage: How many victims does the system identified out of all victims under the debris. If the system discovered 6 victims out of 10 victims actually under the debris, the system has 60% coverage rate.
Accuracy: How many identification were correct out of all identifications made by the system. If the system spotted 8 places as victims, and actually only 4 of them are correct, accuracy is 50%.
Rescue Ratio: How many victims discovered were actually saved? If the system found 10 victims, but are only able to save 5 of them, the rescue ratio is 50%.
Effective Rescue Ratio: How many victims out of all victims under the debris, the system finally saved?
Operational Loading: Numbers of operations that has to made by each human operator. This measure evaluate how many equipments can be operated at the time of disaster where human resource is bottleneck.

Overall evaluation function will be provided shortly to linealize the evaluation criteira above.

Multiple Site Senario

Within a few years, the senario can be extended to cope with multiple disaster sites using multiple rescue teams that involves high-level decision and appropriate resource allocations.