光学头部姿态跟踪的多传感器数据融合研究

精确的头部姿态跟踪是室内增强现实系统实现高精度注册的关键技术之一. 本文介绍了使用传感器数据融合原理实现高精度的光学头部姿态跟踪的新方法. 该方法使用多传感器数据融合中的扩展卡尔曼滤波器和融合滤波器, 将两个互补的单摄像机Inside-out跟踪和双摄像机Outside-in跟踪的头部姿态进行数据融合, 以减小光学跟踪传感器的姿态误差. 设计了一个典型实验装置验证所提出的算法, 实验结果显示, 在静态测试下的姿态输出误差与使用误差协方差传播法则计算得到的结果是一致的; 在动态跟踪条件下, 与单个Inside-out或Outside-in跟踪相比, 所提出的光学头部姿态数据融合算法能够使跟踪器获得精度更高、更稳定的位置和方向信息.     

A low-complexity sensor fusion algorithm based on a fiber-optic gyroscope aided camera pose estimation system

Visual tracking, as a popular computer vision technique, has a wide range of applications, such as camera pose estimation. Conventional methods for it are mostly based on vision only, which are complex for image processing due to the use of only one sensor. This paper proposes a novel sensor fusion algorithm fusing the data from the camera and the fiber-optic gyroscope. In this system, the camera acquires images and detects the object directly at the beginning of each tracking stage; while the relative motion between the camera and the object measured by the fiber-optic gyroscope can track the object coordinate so that it can improve the effectiveness of visual tracking. Therefore, the sensor fusion algorithm presented based on the tracking system can overcome the drawbacks of the two sensors and take advantage of the sensor fusion to track the object accurately. In addition, the computational complexity of our proposed algorithm is obviously lower compared with the existing approaches(86% reducing for a 0.5 min visual tracking). Experiment results show that this visual tracking system reduces the tracking error by 6.15% comparing with the conventional vision-only tracking scheme(edge detection), and our proposed sensor fusion algorithm can achieve a long-term tracking with the help of bias drift suppression calibration.     

Hierarchical On-line Appearance-Based Tracking for 3D head pose,eyebrows, lips,eyelids and irises

In this paper, we propose an On-line Appearance-Based Tracker (OABT) for simultaneous tracking of 3D head pose, lips, eyebrows, eyelids and irises in monocular video sequences. In contrast to previously proposed tracking approaches, which deal with face and gaze tracking separately, our OABT can also be used for eyelid and iris tracking, as well as 3D head pose, lips and eyebrows facial actions tracking. Furthermore, our approach applies an on-line learning of changes in the appearance of the tracked target. Hence, the prior training of appearance models, which usually requires a large amount of labeled facial images, is avoided. Moreover, the proposed method is built upon a hierarchical combination of three OABTs, which are optimized using a Levenberg–Marquardt Algorithm (LMA) enhanced with line-search procedures. This, in turn, makes the proposed method robust to changes in lighting conditions, occlusions and translucent textures, as evidenced by our experiments. Finally, the proposed method achieves head and facial actions tracking in real-time.     

A Flexible Software Architecture for Hybrid Tracking

Fusion of vision-based and inertial pose estimation has many high-potential applications in navigation, robotics, and augmented reality. Our research aims at the development of a fully mobile, completely self-contained tracking system, that is able to estimate sensor motion from known 3D scene structure. This requires a highly modular and scalable software architecture for algorithm design and testing. As the main contribution of this paper, we discuss the design of our hybrid tracker and emphasize important features: scalability, code reusability, and testing facilities. In addition, we present a mobile augmented reality application, and several first experiments with a fully mobile vision-inertial sensor head. Our hybrid tracking system is not only capable of real-time performance, but can also be used for offline analysis of tracker performance, comparison with ground truth, and evaluation of several pose estimation and information fusion algorithms. © 2004 Wiley Periodicals, Inc.     

Real-time view-based pose recognition and interpolation for tracking initialization

In this paper we propose a new approach to real-time view-based pose recognition and interpolation. Pose recognition is particularly useful for identifying camera views in databases, video sequences, video streams, and live recordings. All of these applications require a fast pose recognition process, in many cases video real-time. It should further be possible to extend the database with new material, i.e., to update the recognition system online. The method that we propose is based on P-channels, a special kind of information representation which combines advantages of histograms and local linear models. Our approach is motivated by its similarity to information representation in biological systems but its main advantage is its robustness against common distortions such as clutter and occlusion. The recognition algorithm consists of three steps: (1) low-level image features for color and local orientation are extracted in each point of the image; (2) these features are encoded into P-channels by combining similar features within local image regions; (3) the query P-channels are compared to a set of prototype P-channels in a database using a least-squares approach. The algorithm is applied in two scene registration experiments with fisheye camera data, one for pose interpolation from synthetic images and one for finding the nearest view in a set of real images. The method compares favorable to SIFT-based methods, in particular concerning interpolation. The method can be used for initializing pose-tracking systems, either when starting the tracking or when the tracking has failed and the system needs to re-initialize. Due to its real-time performance, the method can also be embedded directly into the tracking system, allowing a sensor fusion unit choosing dynamically between the frame-by-frame tracking and the pose recognition.     

Cooperative localization for disconnected sensor networks and a mobile robot in friendly environments

Localization for a disconnected sensor network is highly unlikely to be achieved by its own sensor nodes, since accessibility of the information between any pair of sensor nodes cannot be guaranteed. In this paper, a mobile robot (or a mobile sensor node) is introduced to establish correlations among sparsely distributed sensor nodes which are disconnected, even isolated. The robot and the sensor network operate in a friendly manner, in which they can cooperate to perceive each other for achieving more accurate localization, rather than trying to avoid being detected by each other. The mobility of the robot allows for the stationary and internally disconnected sensor nodes to be dynamically connected and correlated. On one hand, the robot performs simultaneous localization and mapping (SLAM) based on the constrained local submap filter (CLSF). The robot creates a local submap composed of the sensor nodes present in its immediate vicinity. The locations of these nodes and the pose (position and orientation angle) of the robot are estimated within the local submap. On the other hand, the sensor nodes in the submap estimate the pose of the robot. A parallax-based robot pose estimation and tracking (PROPET) algorithm, which uses the relationship between two successive measurements of the robot's range and bearing, is proposed to continuously track the robot's pose with each sensor node. Then, tracking results of the robot's pose from different sensor nodes are fused by the Kalman filter (KF). The multi-node fusion result are further integrated with the robot's SLAM result within the local submap to achieve more accurate localization for the robot and the sensor nodes. Finally, the submap is projected and fused into the global map by the CLSF to generate localization results represented in the global frame of reference. Simulation and experimental results are presented to show the performances of the proposed method for robot-sensor network cooperative localization. Especially, if the robot (or the mobile sensor node) has the same sensing ability as the stationary sensor nodes, the localization accuracy can be significantly enhanced using the proposed method.     

Analysis of head pose accuracy in augmented reality

A method is developed to analyze the accuracy of the relative head-to-object position and orientation (pose) in augmented reality systems with head-mounted displays. From probabilistic estimates of the errors in optical tracking sensors, the uncertainty in head-to-object pose can be computed in the form of a covariance matrix. The positional uncertainty can be visualized as a 3D ellipsoid. One useful benefit of having an explicit representation of uncertainty is that we can fuse sensor data from a combination of fixed and head-mounted sensors in order to improve the overall registration accuracy. The method was applied to the analysis of an experimental augmented reality system, incorporating an optical see-through head-mounted display, a head-mounted CCD camera, and a fixed optical tracking sensor. The uncertainty of the pose of a movable object with respect to the head-mounted display was analyzed. By using both fixed and head mounted sensors, we produced a pose estimate that is significantly more accurate than that produced by either sensor acting alone     

基于贝叶斯理论框架的传感器选择算法

针对大规模传感器网络(Large-scale sensor networks)的目标跟踪问题, 本文在贝叶斯(Bayes)框架下, 提出了一种全新的基于传感器选择的多传感器目标跟踪算法.算法的具体思路为:首先基于Bayes框架, 根据不同的管理目标, 推导出传感器选择的目标函数; 然后根据目标函数, 计算出相应的传感器选择方案; 最后将选择的传感器进行数据融合, 求得传感器网络的目标跟踪结果.相比传统的基于量测野值点剔除思想的目标跟踪算法以及基于系统偏差估计的传感器配准算法, 本文提出的基于传感器选择的多传感器目标跟踪算法不仅目标跟踪精度更高, 且跟踪性能更稳定.同时本文提出的传感器选择算法还可以适用于杂波数目较少的目标跟踪场景.仿真结果说明了本文所提算法的有效性.     

Heterogeneous multisensor fusion for mapping dynamic environments

In this paper, we propose a heterogeneous multisensor fusion algorithm for mapping in dynamic environments. The algorithm synergistically integrates the information obtained from an uncalibrated camera and sonar sensors to facilitate mapping and tracking. The sonar data is mainly used to build a weighted line-based map via the fuzzy clustering technique. The line weight, with confidence corresponding to the moving object, is determined by both sonar and vision data. The motion tracking is primarily accomplished by vision data using particle filtering and the sonar vectors originated from moving objects are used to modulate the sample weighting. A fuzzy system is implemented to fuse the two sensor data features. Additionally, in order to build a consistent global map and maintain reliable tracking of moving objects, the well-known extended Kalman filter is applied to estimate the states of robot pose and map features. Thus, more robust performance in mapping as well as tracking are achieved. The empirical results carried out on the Pioneer 2DX mobile robot demonstrate that the proposed algorithm outperforms the methods a using homogeneous sensor, in mapping as well as tracking behaviors.     

基于头部姿态分析的摄像头视线追踪系统优化

实时视线跟踪技术是智能眼动操作系统的关键技术。与基于眼动仪的技术相比,基于网络摄像头的技术具有低成本、高通用性等优点。针对现有的基于摄像头的算法只考虑眼部图像特征、准确度较低的问题,提出引入头部姿态分析的视线追踪算法优化技术。首先,通过人脸特征点检测结果构建头部姿态特征,为标定数据提供头部姿态上下文;然后,研究了新的相似度算法,计算头部姿态上下文的相似度;最后,在进行视线追踪时,利用头部姿态相似度对校准数据进行过滤,从标定数据集中选取与当前输入帧头部姿态相似度较高的数据进行预测。在选取不同特征人群的数据上进行了大量实验,对比实验结果显示,与WebGazer相比,所提算法的平均误差降低了58~63 px。所提算法能有效提高追踪结果的准确性和稳定性,拓展了摄像头设备在视线追踪领域的应用场景。     

基于头部姿态分析的摄像头视线追踪系统优化

实时视线跟踪技术是智能眼动操作系统的关键技术。与基于眼动仪的技术相比，基于网络摄像头的技术具有低成本、高通用性等优点。针对现有的基于摄像头的算法只考虑眼部图像特征、准确度较低的问题，提出引入头部姿态分析的视线追踪算法优化技术。首先，通过人脸特征点检测结果构建头部姿态特征，为标定数据提供头部姿态上下文；然后，研究了新的相似度算法，计算头部姿态上下文的相似度；最后，在进行视线追踪时，利用头部姿态相似度对校准数据进行过滤，从标定数据集中选取与当前输入帧头部姿态相似度较高的数据进行预测。在选取不同特征人群的数据上进行了大量实验，对比实验结果显示，与WebGazer相比，所提算法的平均误差降低了58~63 px。所提算法能有效提高追踪结果的准确性和稳定性，拓展了摄像头设备在视线追踪领域的应用场景。     

Multi‐sensor track‐to‐track fusion via linear minimum variance sense estimators

An integrated approach that consists of sensor‐based filtering algorithms, local processors, and a global processor is employed to describe the distributed fusion problem when several sensors execute surveillance over a certain area. For the sensor tracking systems, each filtering algorithm utilized in the reference Cartesian coordinate system is presented for target tracking, with the radar measuring range, bearing, and elevation angle in the spherical coordinate system (SCS). For the local processors, each track‐to‐track fusion algorithm is used to merge two tracks representing the same target. The number of 2‐combinations of a set with N distinct sensors is considered for central track fusion. For the global processor, the data fusion algorithms, simplified maximum likelihood (SML) estimator and covariance matching method (CMM), based on linear minimum variance (LMV) estimation fusion theory, are developed for use in a centralized track‐to‐track fusion situation. The resulting global fusers can be implemented in a parallel structure to facilitate estimation fusion calculation. Simulation results show that the proposed SML estimator has a more robust capability of improving tracking accuracy than the CMM and the LMV estimators. Copyright © 2008 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

An information fusion framework for person localization via body pose in spectator crowds

Person localization or segmentation in low resolution crowded scenes is important for person tracking and recognition, action detection and anomaly identification. Due to occlusion and lack of inter-person space, person localization becomes a difficult task. In this work, we propose a novel information fusion framework to integrate a Deep Head Detector and a body pose detector. A more accurate body pose showing limb positions will result in more accurate person localization. We propose a novel Deep Head Detector (DHD) to detect person heads in crowds. The proposed DHD is a fully convolutional neural network and it has shown improved head detection performance in crowds. We modify Deformable Parts Model (DPM) pose detector to detect multiple upper body poses in crowds. We efficiently fuse the information obtained by the proposed DHD and the modified DPM to obtain a more accurate person pose detector. The proposed framework is named as Fusion DPM (FDPM) and it has exhibited improved body pose detection performance on spectator crowds. The detected body poses are then used for more accurate person localization by segmenting each person in the crowd.     

Hybrid three-dimensional camera pose estimation using particle filter sensor fusion

To properly align objects in the real and virtual worlds in an augmented reality (AR) space it is essential to keep tracking the camera's exact three-dimensional position and orientation (camera pose). State-of-the-art analysis shows that traditional vision-based or inertial sensor-based solutions are not adequate when used individually. Sensor fusion for hybrid tracking has become an active research direction during the past few years, although how to do it in a robust and principled way is still an open problem. In this paper, we develop a hybrid camera pose-tracking system that combines vision and inertial sensor technologies. We propose to use the particle filter framework for the sensor fusion system. Particle filters are sequential Monte-Carlo methods based upon a point mass (or 'particle') representation of probability densities, which can be applied to any state space model and which generalize the traditional Kalman filtering methods. We have tested our algorithm to evaluate its performance and have compared the results obtained by the particle filter with those given by a classical extended Kalman filter. Experimental results are presented     

Robust real-time 3D head pose estimation from range data

In this paper a real-time 3D pose estimation algorithm using range data is described. The system relies on a novel 3D sensor that generates a dense range image of the scene. By not relying on brightness information, the proposed system guarantees robustness under a variety of illumination conditions, and scene contents. Efficient face detection using global features and exploitation of prior knowledge along with novel feature localization and tracking techniques are described. Experimental results demonstrate accurate estimation of the six degrees of freedom of the head and robustness under occlusions, facial expressions, and head shape variability.     

Scale-aware navigation of a low-cost quadrocopter with a monocular camera

We present a complete solution for the visual navigation of a small-scale, low-cost quadrocopter in unknown environments. Our approach relies solely on a monocular camera as the main sensor, and therefore does not need external tracking aids such as GPS or visual markers. Costly computations are carried out on an external laptop that communicates over wireless LAN with the quadrocopter. Our approach consists of three components: a monocular SLAM system, an extended Kalman filter for data fusion, and a PID controller. In this paper, we (1) propose a simple, yet effective method to compensate for large delays in the control loop using an accurate model of the quadrocopter’s flight dynamics, and (2) present a novel, closed-form method to estimate the scale of a monocular SLAM system from additional metric sensors. We extensively evaluated our system in terms of pose estimation accuracy, flight accuracy, and flight agility using an external motion capture system. Furthermore, we compared the convergence and accuracy of our scale estimation method for an ultrasound altimeter and an air pressure sensor with filtering-based approaches. The complete system is available as open-source in ROS. This software can be used directly with a low-cost, off-the-shelf Parrot AR.Drone quadrocopter, and hence serves as an ideal basis for follow-up research projects.     

Optimized data rate allocation for dynamic sensor fusion over resource constrained communication networks

This article presents a new method to solve a dynamic sensor fusion problem. We consider a large number of remote sensors which measure a common Gauss–Markov process. Each sensor encodes and transmits its measurement to a data fusion center through a resource restricted communication network. The communication cost incurred by a given sensor is quantified as the expected bitrate from the sensor to the fusion center. We propose an approach that attempts to minimize a weighted sum of these communication costs subject to a constraint on the state estimation error at the fusion center. We formulate the problem as a difference-of-convex program and apply the convex-concave procedure (CCP) to obtain a heuristic solution. We consider a 1D heat transfer model and a model for 2D target tracking by a drone swarm for numerical studies. Through these simulations, we observe that our proposed approach has a tendency to assign zero data rate to unnecessary sensors indicating that our approach is sparsity-promoting, and an effective sensor selection heuristic.     

Recognizing Visual Focus of Attention From Head Pose in Natural Meetings

We address the problem of recognizing the visual focus of attention (VFOA) of meeting participants based on their head pose. To this end, the head pose observations are modeled using a Gaussian mixture model (GMM) or a hidden Markov model (HMM) whose hidden states correspond to the VFOA. The novelties of this paper are threefold. First, contrary to previous studies on the topic, in our setup, the potential VFOA of a person is not restricted to other participants only. It includes environmental targets as well (a table and a projection screen), which increases the complexity of the task, with more VFOA targets spread in the pan as well as tilt gaze space. Second, we propose a geometric model to set the GMM or HMM parameters by exploiting results from cognitive science on saccadic eye motion, which allows the prediction of the head pose given a gaze target. Third, an unsupervised parameter adaptation step not using any labeled data is proposed, which accounts for the specific gazing behavior of each participant. Using a publicly available corpus of eight meetings featuring four persons, we analyze the above methods by evaluating, through objective performance measures, the recognition of the VFOA from head pose information obtained either using a magnetic sensor device or a vision-based tracking system. The results clearly show that in such complex but realistic situations, the VFOA recognition performance is highly dependent on how well the visual targets are separated for a given meeting participant. In addition, the results show that the use of a geometric model with unsupervised adaptation achieves better results than the use of training data to set the HMM parameters.     

An integrated head pose and eye gaze tracking approach to non-intrusive visual attention measurement for wide FOV simulators

Eye gaze tracking is very useful for quantitatively measuring visual attention in virtual environments. However, most eye trackers have a limited tracking range, e.g., ±35° in the horizontal direction. This paper proposed a method to combine head pose tracking and eye gaze tracking together to achieve a large range of tracking in virtual driving simulation environments. Multiple parallel multilayer perceptrons were used to reconstruct the relationship between head images and head poses. Head images were represented with the coefficients extracted from Principal Component Analysis. Eye gaze tracking provides precise results on the front view, while head pose tracking is more suitable for tracking areas of interest than for tracking points of interest on the side view.     

A multi-cue mean-shift target tracking approach based on fuzzified region dynamic image fusion

Traditional target tracking algorithms based on single sensor images are unstable and have low accuracy. Based on regional target detection and fuzzy region rules, a fuzzy region-based multi-sensor image fusion approach is proposed in this paper. The similarity measure weight is adapted to this dynamic image fusion algorithm, while the tracking method uses the proposed multi-cue mean-shift tracking algorithm. Three experimental results using real world image sequences are evaluated using the steady state square root mean error. The fusion and tracking experiments indicate that the proposed approach is effective and efficient when aiming at a target moving from one area to a different area, which meets the robustness and real-time requirements.