Thermal image-based hand gesture recognition for worker-robot collaboration in the construction industry: A feasible study

Worker-robot collaboration (WRC) is a promising solution for complex construction tasks, which can integrate the robots’ advantages in strength and accuracy with human ability in intuitive decision-making and adaptability. A new imperative objective for real-world WRC is to design a user-friendly interface to support safe and efficient worker-robot interactions. Vision-based hand gesture is a simple but effective solution. However, existing methods mainly depend on 3-channel RGB images captured by visible cameras, which are prone to be affected by on-site environmental disturbances, such as poor illumination, fog, and dust. Moreover, previous networks strive for high accuracy, neglecting computational efficiency (e.g., model size and latency) when implementing the network on resource-constrained devices like mobile construction robots. Against this backdrop, this research presented a feasibility study to investigate whether hand signals can be detected from thermal images and designed a lightweight network that has fewer parameters and obtains lower latency without compromising accuracy. Experimental results indicated that thermal images were robust to different lighting conditions, and the proposed model achieved a high classification accuracy (97.54 %) with 1.8 M parameters. The comparative study demonstrated the superiority of our model to other advanced lightweight models, illustrating the feasibility of the developed method in supporting safe WRC applications by recognizing workers’ hand signals.