Aerial online mapping on-board system by real-time object detection for UGV path generation in unstructured outdoor environments

An optimal path provides efficient operation of unmanned ground vehicles (UGVs) for many kinds of tasks such as transportation, exploration, surveillance, and search and rescue in unstructured areas that include various unexpected obstacles. Various onboard sensors such as LiDAR, radar, sonar, and cameras are used to detect obstacles around the UGVs. However, their range of view is often limited by movable obstacles or barriers, resulting in inefficient path generation. Here, we present the aerial online mapping system to generate an efficient path for a UGV on a two-dimensional map. The map is updated by projecting obstacles detected in the aerial images taken by an unmanned aerial vehicle through an object detector based on a conventional convolutional neural network. The proposed system is implemented in real-time by a skid steering ground vehicle and a quadcopter with relatively small, low-cost embedded systems. The frameworks and each module of the systems are given in detail to evaluate the performance. The system is also demonstrated in unstructured outdoor environments such as in a football field and a park with unreliable communication links. The results show that the aerial online mapping is effective in path generation for autonomous UGVs in real environments.     

Real-time Implementation and Validation of a New Hierarchical Path Planning Scheme of UAVs via Hardware-in-the-Loop Simulation

We present a real-time hardware-in-the-loop simulation environment for the validation of a new hierarchical path planning and control algorithm for a small fixed-wing unmanned aerial vehicle (UAV). The complete control algorithm is validated through on-board, real-time implementation on a small autopilot having limited computational resources. We present two distinct real-time software frameworks for implementing the overall control architecture, including path planning, path smoothing, and path following. We emphasize, in particular, the use of a real-time kernel, which is shown to be an effective and robust way to accomplish real-time operation of small UAVs under non-trivial scenarios. By seamless integration of the whole control hierarchy using the real-time kernel, we demonstrate the soundness of the approach. The UAV equipped with a small autopilot, despite its limited computational resources, manages to accomplish sophisticated unsupervised navigation to the target, while autonomously avoiding obstacles.     

Navigation Aided Image Processing in UAV Surveillance: Preliminary Results and Design of an Airborne Experimental System

This paper describes an airborne reconfigurable measurement system being developed at Swedish Defence Research Agency (FOI), Sensor Technology, Sweden. An image processing oriented sensor management architecture for UAV (unmanned aerial vehicles) IR/EO-surveillance is presented. Some preliminary results of navigation aided image processing in UAV applications are demonstrated, such as SLAM (simultaneous localization and mapping), structure from motion and geolocation, target tracking, and detection of moving objects. The design goal of the measurement system is to emulate a UAV-mounted sensor gimbal using a stand-alone system. The minimal configuration of the system consists of a gyro-stabilized gimbal with IR and CCD sensors and an integrated high-performance navigation system. The navigation system combines dGPS real-time kinematics (RTK) data with data from an inertial measurement unit (IMU) mounted with reference to the optical sensors. The gimbal is to be used as an experimental georeferenced sensor platform, using a choice of carriers, to produce military relevant image sequences for studies of image processing and sensor control on moving surveillance and reconnaissance platforms. Furthermore, a high resolution synthetic environment, developed for sensor simulations in the visual and infrared wavelengths, is presented. © 2004 Wiley Periodicals, Inc.     

Collision-free Autonomous Navigation of A Small UAV Using Low-cost Sensors in GPS-denied Environments

This paper proposes a novel complete navigation system for autonomous flight of small unmanned aerial vehicles (UAVs) in GPS-denied environments. The hardware platform used to test the proposed algorithm is a small, custom-built UAV platform equipped with an onboard computer, RGB-D camera, 2D light detection and ranging (LiDAR), and altimeter. The error-state Kalman filter (ESKF) based on the dynamic model for low-cost IMU-driven systems is proposed, and visual odometry from the RGB-D camera and height measurement from the altimeter are fed into the measurement update process of the ESKF. The pose output of the ESKF is then integrated into the open-source simultaneous location and mapping (SLAM) algorithm for pose-graph optimization and loop closing. In addition, the computationally efficient collision-free path planning algorithm is proposed and verified through simulations. The software modules run onboard in real time with limited onboard computational capability. The indoor flight experiment demonstrates that the proposed system for small UAVs with low-cost devices can navigate without collision in fully autonomous missions while establishing accurate surrounding maps.

     

Autonomous Navigation of UAV in Foliage Environment

This paper presents a navigation system that enables small-scale unmanned aerial vehicles to navigate autonomously using a 2D laser range finder in foliage environment without GPS. The navigation framework consists of real-time dual layer control, navigation state estimation and online path planning. In particular, the inner loop of a quadrotor is stabilized using a commercial autopilot while the outer loop control is implemented using robust perfect tracking. The navigation state estimation consists of real-time onboard motion estimation and trajectory smoothing using the GraphSLAM technique. The onboard real-time motion estimation is achieved by a Kalman filter, fusing the planar velocity measurement from matching the consecutive scans of a laser range finder and the acceleration measurement of an inertial measurement unit. The trajectory histories from the real-time autonomous navigation together with the observed features are fed into a sliding-window based pose-graph optimization framework. The online path planning module finds an obstacle-free trajectory based the local measurement of the laser range finder. The performance of the proposed navigation system is demonstrated successfully on the autonomous navigation of a small-scale UAV in foliage environment.     

鱼雷状小型无人艇路径跟踪控制系统研制

为了提高无人艇在水域作业时的航向跟踪精度,自主设计了一款鱼雷状小型无人艇路径跟踪控制系统。该系统由岸基控制系统和艇载控制系统组成,具有自主巡航和手动控制2种工作模式。岸基控制系统通过数传电台与艇载控制系统进行信息交互,显示传回的状态信息并下达控制指令,在自主航行模式下完成BD09与WGS84的坐标系转换、期望航向角计算和目标点更新;艇载控制系统采用STM32F429作为控制芯片,完成数据采集和运动控制,在自主巡航模式下提供位置和航向数据,通过路径跟踪控制器输出的舵机PWM信号调整航向。实验表明设计的路径跟踪控制系统运行稳定,精度较高。     

Development of a vision-based ground target detection and tracking system for a small unmanned helicopter

It is undoubted that the latest trend in the unmanned aerial vehicles (UAVs) community is towards visionbased unmanned small-scale helicopter, utilizing the maneuvering capabilities of the helicopter and the rich information of visual sensors, in order to arrive at a versatile platform for a variety of applications such as navigation, surveillance, tracking, etc. In this paper, we present the development of a visionbased ground target detection and tracking system for a small UAV helicopter. More specifically, we propose a real-time vision algorithm, based on moment invariants and two-stage pattern recognition, to achieve automatic ground target detection. In the proposed algorithm, the key geometry features of the target are extracted to detect and identify the target. Simultaneously, a Kalman filter is used to estimate and predict the position of the target, referred to as dynamic features, based on its motion model. These dynamic features are then combined with geometry features to identify the target in the second-stage of pattern recognition, when geometry features of the target change significantly due to noise and disturbance in the environment. Once the target is identified, an automatic control scheme is utilized to control the pan/tilt visual mechanism mounted on the helicopter such that the identified target is to be tracked at the center of the captured images. Experimental results based on images captured by the small-scale unmanned helicopter, SheLion, in actual flight tests demonstrate the effectiveness and robustness of the overall system.     

Vision-based navigation of unmanned aerial vehicles

This paper presents a vision-based navigation strategy for a vertical take-off and landing (VTOL) unmanned aerial vehicle (UAV) using a single embedded camera observing natural landmarks. In the proposed approach, images of the environment are first sampled, stored and organized as a set of ordered key images (visual path) which provides a visual memory of the environment. The robot navigation task is then defined as a concatenation of visual path subsets (called visual route) linking the current observed image and a target image belonging to the visual memory. The UAV is controlled to reach each image of the visual route using a vision-based control law adapted to its dynamic model and without explicitly planning any trajectory. This framework is largely substantiated by experiments with an X4-flyer equipped with a fisheye camera.     

Conflict-free navigation in unknown urban environments

This paper presents an autonomous exploration method in an unknown environment that uses model predictive control (MPC)-based obstacle avoidance with local map building by onboard sensing. An onboard laser scanner is used to build an online map of obstacles around the vehicle with outstanding accuracy. This local map is combined with a real-time MPC algorithm that generates a safe vehicle path, using a cost function that penalizes the proximity to the nearest obstacle. The adjusted trajectory is then sent to a position tracking layer in the hierarchical unmanned aerial vehicle (UAV) avionics architecture. In a series of experiments using a Berkeley UAV, the proposed approach successfully guided the vehicle safely through the urban canyon.     

Bearing-only visual SLAM for small unmanned aerial vehicles in GPS-denied environments

This paper presents a hierarchical simultaneous localization and mapping(SLAM) system for a small unmanned aerial vehicle(UAV) using the output of an inertial measurement unit(IMU) and the bearing-only observations from an onboard monocular camera.A homography based approach is used to calculate the motion of the vehicle in 6 degrees of freedom by image feature match.This visual measurement is fused with the inertial outputs by an indirect extended Kalman filter(EKF) for attitude and velocity estimation.Then,another EKF is employed to estimate the position of the vehicle and the locations of the features in the map.Both simulations and experiments are carried out to test the performance of the proposed system.The result of the comparison with the referential global positioning system/inertial navigation system(GPS/INS) navigation indicates that the proposed SLAM can provide reliable and stable state estimation for small UAVs in GPS-denied environments.     

基于xPC的视觉制导半实物仿真快速开发方法

视觉传感器作为无人飞行器的一种重要传感器,对无人飞行器的制导与控制起着至关重要的作用.研究以视觉传感器为对象的飞行控制系统半实物仿真,对于了解视觉制导的工作机理和制导控制方法有着重要的意义.研究了一种基于导航制导与控制实验系统的视觉传感器半实物仿真快速开发方法.采用快速代码生成技术和接口模块库的快速开发方法,可以大大提高验证制导控制算法优劣性的效率,为研究基于视觉传感器的制导控制方法提供一个简洁快速的验证平台.     

Automatic Take Off, Tracking and Landing of a Miniature UAV on a Moving Carrier Vehicle

We present a system consisting of a miniature unmanned aerial vehicle (UAV) and a small carrier vehicle, in which the UAV is capable of autonomously starting from the moving ground vehicle, tracking it at a constant distance and landing on a platform on the carrier in motion. Our visual tracking approach differs from other methods by using low-cost, lightweight commodity consumer hardware. As main sensor we use a Wii remote infrared (IR) camera, which allows robust tracking of a pattern of IR lights in conditions without direct sunlight. The system does not need to communicate with the ground vehicle and works with an onboard 8-bit microcontroller. Nevertheless the position and orientation relative to the IR pattern is estimated at a frequency of approximately 50 Hz. This enables the UAV to fly fully autonomously, performing flight control, self-stabilisation and visual tracking of the ground vehicle. We present experiments in which our UAV performs autonomous flights with a moving ground carrier describing a circular path and where the carrier is rotating. The system provides small errors and allows for safe, autonomous indoor flights.     

一套完整的基于视觉光流和激光扫描测距的室内无人机导航系统

提出了一套室内四旋翼无人机控制, 导航, 定位和地图构建的完整解决方案. 无人机机载系统包括三个主要传感器, 即惯性测量单元, 下视相机和激光扫描测距仪. 经过处理, 融合这些传感器的测量数据, 无人机能够可靠的估计自己的飞行速度和实时位置, 并且沿着室内的墙壁进行无碰撞飞行. 通过收集一个完整飞行实验的数据, 无人机的飞行路径和在室内的环境也可以被很好地估计出来. 这套系统中的自主导功能不需要任何远程传感信息或脱机计算能力. 这套室内导航方案的性能和可靠性已在实际的飞行实验中被验证.     

改进YOLOv3在航拍目标检测中的应用

近年来,基于深度学习的航拍目标检测在无人驾驶、军事侦察、灾害检测等领域有着广泛的应用,更精确、高效的算法是目前航拍目标检测研究的热点与难点。提出一种基于改进YOLOv3算法的航拍目标检测方法,对航拍数据集进行目标先验框维度聚类、优化锚点参数,提高了网络对航拍目标的检测有效性。同时对原网络进行改进,减少部分卷积操作并引入跳跃连接机制降低特征冗余,提高了检测准确率,并降低了小目标的误检率与漏检率。实验结果表明,改进YOLOv3算法相较于原始YOLOv3算法的效果有明显提升,对于较高分辨率的航拍图像,加快了网络的收敛速度,并在保证实时性的前提下,将检测平均准确率（mean Average Precision,mAP）提高了12.7%。     

An on-board vision sensor system for small unmanned vehicle applications

This paper describes an on-board vision sensor system that is developed specifically for small unmanned vehicle applications. For small vehicles, vision sensors have many advantages, including size, weight, and power consumption, over other sensors such as radar, sonar, and laser range finder, etc. A vision sensor is also uniquely suited for tasks such as target tracking and recognition that require visual information processing. However, it is difficult to meet the computing needs of real-time vision processing on a small robot. In this paper, we present the development of a field programmable gate array-based vision sensor and use a small ground vehicle to demonstrate that this vision sensor is able to detect and track features on a user-selected target from frame to frame and steer the small autonomous vehicle towards it. The sensor system utilizes hardware implementations of the rank transform for filtering, a Harris corner detector for feature detection, and a correlation algorithm for feature matching and tracking. With additional capabilities supported in software, the operational system communicates wirelessly with a base station, receiving commands, providing visual feedback to the user and allowing user input such as specifying targets to track. Since this vision sensor system uses reconfigurable hardware, other vision algorithms such as stereo vision and motion analysis can be implemented to reconfigure the system for other real-time vision applications.     

Low-Cost Visual Tracking of a Landing Place and Hovering Flight Control with a Microcontroller

The growth of civil and military use has recently promoted the development of unmanned miniature aerial vehicles dedicated to surveillance tasks. These flying vehicles are often capable of carrying only a few dozen grammes of payload. To achieve autonomy for this kind of aircraft novel sensors are required, which need to cope with strictly limited onboard processing power. One of the key aspects in autonomous behaviour is target tracking. Our visual tracking approach differs from other methods by not using expensive cameras but a Wii remote camera, i.e. commodity consumer hardware. The system works without stationary sensors and all processing is done with an onboard microcontroller. The only assumptions are a good roll and pitch attitude estimation, provided by an inertial measurement unit and a stationary pattern of four infrared spots on the target or the landing spot. This paper details experiments for hovering above a landing place, but tracking a slowly moving target is also possible.     

Autonomous Vision-Based Helicopter Flights Through Obstacle Gates

The challenge for unmanned aerial vehicles to sense and avoid obstacles becomes even harder if narrow passages have to be crossed. An approach to solve a mission scenario that tackles the problem of such narrow passages is presented here. The task is to fly an unmanned helicopter autonomously through a course with gates that are only slightly larger than the vehicle itself. A camera is installed on the vehicle to detect the gates. Using vehicle localization data from a navigation solution, camera alignment and global gate positions are estimated simultaneously. The presented algorithm calculates the desired target waypoints to fly through the gates. Furthermore, the paper presents a mission execution plan that instructs the vehicle to search for a gate, to fly through it after successful detection, and to search for a proceeding one. All algorithms are designed to run onboard the vehicle so that no interaction with the ground control station is necessary, making the vehicle completely autonomous. To develop and optimize algorithms, and to prove the correctness and accuracy of vision-based gate detection under real operational conditions, gate positions are searched in images taken from manual helicopter flights. Afterwards, the integration of visual sensing and mission control is proven. The paper presents results from full autonomous flight where the helicopter searches and flies through a gate without operator actions.     

A Practical Visual Servo Control for an Unmanned Aerial Vehicle

An image-based visual servo control is presented for an unmanned aerial vehicle (UAV) capable of stationary or quasi-stationary flight with the camera mounted onboard the vehicle. The target considered consists of a finite set of stationary and disjoint points lying in a plane. Control of the position and orientation dynamics is decoupled using a visual error based on spherical centroid data, along with estimations of the linear velocity and the gravitational inertial direction extracted from image features and an embedded inertial measurement unit. The visual error used compensates for poor conditioning of the image Jacobian matrix by introducing a nonhomogeneous gain term adapted to the visual sensitivity of the error measurements. A nonlinear controller, that ensures exponential convergence of the system considered, is derived for the full dynamics of the system using control Lyapunov function design techniques. Experimental results on a quadrotor UAV, developed by the French Atomic Energy Commission, demonstrate the robustness and performance of the proposed control strategy.     

UAV Mobile Ground Target Pursuit Algorithm

In this paper, we propose a comprehensive ground target pursuit algorithm for fixed-wing unmanned aerial vehicles (UAVs). Based on the division of two kinds of possible path patterns, the method generates waypoints step by step and steers the UAV to the latest waypoint. When it is time for waypoint correction, the tracking error will be recorded, and this error will be transferred to the next waypoint for synchronization purposes. An alarm mechanism is applied in case the target moves out of range of the UAV sensor. The noise in the measurement of the target parameters is considered and is processed by a least square estimator. Simulations with three different scenarios are performed, and the results demonstrate that the algorithm is stable, reliable, and computationally efficient.     

Indoor UAV Control Using Multi-Camera Visual Feedback

This paper presents the control of an indoor unmanned aerial vehicle (UAV) using multi-camera visual feedback. For the autonomous flight of the indoor UAV, instead of using onboard sensor information, visual feedback concept is employed by the development of an indoor flight test-bed. The indoor test-bed consists of four major components: the multi-camera system, ground computer, onboard color marker set, and quad-rotor UAV. Since the onboard markers are attached to the pre-defined location, position and attitude of the UAV can be estimated by marker detection algorithm and triangulation method. Additionally, this study introduces a filter algorithm to obtain the full 6-degree of freedom (DOF) pose estimation including velocities and angular rates. The filter algorithm also enhances the performance of the vision system by making up for the weakness of low cost cameras such as poor resolution and large noise. Moreover, for the pose estimation of multiple vehicles, data association algorithm using the geometric relation between cameras is proposed in this paper. The control system is designed based on the classical proportional-integral-derivative (PID) control, which uses the position, velocity and attitude from the vision system and the angular rate from the rate gyro sensor. This paper concludes with both ground and flight test results illustrating the performance and properties of the proposed indoor flight test-bed and the control system using the multi-camera visual feedback.