Long future frame prediction using optical flow-informed deep neural networks for enhancement of robotic teleoperation in high latency environments

High latency in teleoperation has a significant negative impact on operator performance. While deep learning has revolutionized many domains recently, it has not previously been applied to teleoperation enhancement. We propose a novel approach to predict video frames deep into the future using neural networks informed by synthetically generated optical flow information. This can be employed in teleoperated robotic systems that rely on video feeds for operator situational awareness. We have used the image-to-image translation technique as a basis for the prediction of future frames. The Pix2Pix conditional generative adversarial network (cGAN) has been selected as a base network. Optical flow components reflecting real-time control inputs are added to the standard RGB channels of the input image. We have experimented with three data sets of 20,000 input images each that were generated using our custom-designed teleoperation simulator with a 500-ms delay added between the input and target frames. Structural Similarity Index Measures (SSIMs) of 0.60 and Multi-SSIMs of 0.68 were achieved when training the cGAN with three-channel RGB image data. With the five-channel input data (incorporating optical flow) these values improved to 0.67 and 0.74, respectively. Applying Fleiss' κ gave a score of 0.40 for three-channel RGB data, and 0.55 for five-channel optical flow-added data. We are confident the predicted synthetic frames are of sufficient quality and reliability to be presented to teleoperators as a video feed that will enhance teleoperation. To the best of our knowledge, we are the first to attempt to reduce the impacts of latency through future frame prediction using deep neural networks.     

Earthwork digital twin for teleoperation of an automated bulldozer in edge dumping

The unstructured construction site with a complex and changeable environment brings challenges to human–machine interaction. In this study, the earthwork digital twin (DT) is proposed to realize accurate and real-time perception, which can support the teleoperation of an automated bulldozer. The overall framework includes anterior-time DT, real-time DT, and posterior-time DT, which integrates three-dimensional modeling, edge detection, and blade trajectory algorithms. Multidimensional heterogeneous data in bulldozer teleoperation can be displayed in real-time to ensure construction safety. Big data during teleoperation of the construction process can be collected, stored, and analyzed. The DT proposed in this study was successfully applied in a major construction project in China, and testing results show its universality, robustness, and advanced performance. The key technologies proposed in this study can be applied to solve the common problems in the construction industry, which is promising for future intelligent construction.     

Development of a search and rescue robot system for the underground building environment

The underground building environment plays an increasingly important role in the construction of modern cities. To deal with possible fires, collapses, and so on, in underground building space, it is a general trend to use rescue robots to replace humans. This paper proposes a dual-robot system solution for search and rescue in an underground building environment. To speed up rescue and search, the two robots focus on different tasks. However, the environmental perception information and location of them are shared. The primary robot is used to quickly explore the environment in a wide range, identify objects, cross difficult obstacles, and so on. The secondary robot is responsible for grabbing, carrying items, clearing obstacles, and so on. In response to the difficulty of rescue caused by unknown scenes, the Lidar, inertial measurement unit and multiview cameras are integrated for large-scale 3D environment mapping. The depth camera detects the objects to be rescued and locate them on the map. A six-degree-of-freedom manipulator with a two-finger gripper is equipped to open doors and clear roadblocks during the rescue. To solve the problem of severe signal attenuation caused by reinforced concrete walls, corners and long-distance transmission, a wireless multinode networking solution is adopted. In the case of a weak wireless signal, the primary robot uses autonomous exploration for environmental perception. Experimental results show the robots' system has high reliability in over-the-horizon maneuvering, teleoperation of the door opening and grasping, object searching, and environmental perception, and can be well applied to underground search and rescue.