Perception,navigation, and manipulation in the team KAUST approach to the MBZIRC ground robotics challenge

The ground robotics challenge in the Mohammed Bin Zayed International Robotics Challenge required a ground vehicle equipped with a robotic arm to autonomously locate a panel, select a proper size wrench among several options mounted on the panel, and use the wrench to rotate a valve. Autonomy was the critical factor in this challenge, which required the teams to devise algorithms that can operate successfully in a semistructured environment without human supervision. This paper presents the approaches taken by team KAUST to meet this challenge, ranging from in‐house hardware designs to algorithm integration and customization. We separated the whole objective into three interconnected tasks: Navigation, perception, and manipulation. For the navigation task, we developed a basic robotic exploration scheme to find the panel front side where the wrenches were present. For the perception task, we integrated common object detection algorithms with neural networks to identify the proper size wrench precisely. For successful manipulation, we designed and built a custom gripper, which was inspired by the common grasping behavior of a human hand under tight clearance conditions. The modular structure of the proposed approach allowed the team to progress in several subtasks simultaneously. However, the interconnection between the subtasks necessitated a reliable integration framework between these modules for effective implementation. We tuned our algorithms in extensive experimental studies and eventually obtained 10 consecutive successful navigation runs, 96% true wrench detection rate, and high success rate in wrench grasping. Furthermore, successful complete tests proved the reliability and repeatability of our system.