Rigid and non-rigid 3D motion estimation from multiview image sequences

Multiview image sequence processing has been the focus of considerable attention in recent literature. This paper presents an efficient technique for object-based rigid and non-rigid 3D motion estimation, applicable to problems occurring in multiview image sequence coding applications. More specifically, a neural network is formed for the estimation of the rigid 3D motion of each object in the scene, using initially estimated 2D motion vectors corresponding to each camera view. Non-linear error minimization techniques are adopted for neural network weight update. Furthermore, a novel technique is also proposed for the estimation of the local non-rigid deformations, based on the multiview camera geometry. Experimental results using both stereoscopic and trinocular camera setups illustrate and evaluate the proposed scheme.     

Constrained registration for motion compensation in atrial fibrillation ablation procedures

Fluoroscopic overlay images rendered from preoperative volumetric data can provide additional anatomical details to guide physicians during catheter ablation procedures for treatment of atrial fibrillation (AFib). As these overlay images are often compromised by cardiac and respiratory motion, motion compensation methods are needed to keep the overlay images in sync with the fluoroscopic images. So far, these approaches have either required simultaneous biplane imaging for 3-D motion compensation, or in case of monoplane X-ray imaging, provided only a limited 2-D functionality. To overcome the downsides of the previously suggested methods, we propose an approach that facilitates a full 3-D motion compensation even if only monoplane X-ray images are available. To this end, we use a training phase that employs a biplane sequence to establish a patient specific motion model. Afterwards, a constrained model-based 2-D/3-D registration method is used to track a circumferential mapping catheter. This device is commonly used for AFib catheter ablation procedures. Based on the experiments on real patient data, we found that our constrained monoplane 2-D/3-D registration outperformed the unconstrained counterpart and yielded an average 2-D tracking error of 0.6 mm and an average 3-D tracking error of 1.6 mm. The unconstrained 2-D/3-D registration technique yielded a similar 2-D performance, but the 3-D tracking error increased to 3.2 mm mostly due to wrongly estimated 3-D motion components in X-ray view direction. Compared to the conventional 2-D monoplane method, the proposed method provides a more seamless workflow by removing the need for catheter model re-initialization otherwise required when the C-arm view orientation changes. In addition, the proposed method can be straightforwardly combined with the previously introduced biplane motion compensation technique to obtain a good trade-off between accuracy and radiation dose reduction.     

Prediction for human motion tracking failures.

We propose a new and effective method of predicting tracking failures and apply it to the robust analysis of gait and human motion. We define a tracking failure as an event and describe its temporal characteristics using a hidden Markov model (HMM). We represent the human body using a three-dimensional, multicomponent structural model, where each component is designed to independently allow the extraction of certain gait variables. To enable a fault-tolerant tracking and feature extraction system, we introduce a single HMM for each element of the structural model, trained on previous examples of tracking failures. The algorithm derives vector observations for each Markov model using the time-varying noise covariance matrices of the structural model parameters. When transformed with a logarithmic function, the conditional output probability of each HMM is shown to have a causal relationship with imminent tracking failures. We demonstrate the effectiveness of the proposed approach on a variety of multiview video sequences of complex human motion.     

Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms

We propose a powerful video filtering algorithm that exploits temporal and spatial redundancy characterizing natural video sequences. The algorithm implements the paradigm of nonlocal grouping and collaborative filtering, where a higher dimensional transform-domain representation of the observations is leveraged to enforce sparsity, and thus regularize the data: 3-D spatiotemporal volumes are constructed by tracking blocks along trajectories defined by the motion vectors. Mutually similar volumes are then grouped together by stacking them along an additional fourth dimension, thus producing a 4-D structure, termed group, where different types of data correlation exist along the different dimensions: local correlation along the two dimensions of the blocks, temporal correlation along the motion trajectories, and nonlocal spatial correlation (i.e., self-similarity) along the fourth dimension of the group. Collaborative filtering is then realized by transforming each group through a decorrelating 4-D separable transform and then by shrinkage and inverse transformation. In this way, the collaborative filtering provides estimates for each volume stacked in the group, which are then returned and adaptively aggregated to their original positions in the video. The proposed filtering procedure addresses several video processing applications, such as denoising, deblocking, and enhancement of both grayscale and color data. Experimental results prove the effectiveness of our method in terms of both subjective and objective visual quality, and show that it outperforms the state of the art in video denoising.     

Robust shape tracking with multiple models in ultrasound images

This paper addresses object tracking in ultrasound images using a robust multiple model tracker. The proposed tracker has the following features: 1) it uses multiple dynamic models to track the evolution of the object boundary, and 2) it models invalid observations (outliers), reducing their influence on the shape estimates. The problem considered in this paper is the tracking of the left ventricle which is known to be a challenging problem. The heart motion presents two phases (diastole and systole) with different dynamics, the multiple models used in this tracker try to solve this difficulty. In addition, ultrasound images are corrupted by strong multiplicative noise which prevents the use of standard deformable models. Robust estimation techniques are used to address this difficulty. The multiple model data association (MMDA) tracker proposed in this paper is based on a bank of nonlinear filters, organized in a tree structure. The algorithm determines which model is active at each instant of time and updates its state by propagating the probability distribution, using robust estimation techniques.     

Markerless real-time 3-D target region tracking by motion backprojection from projection images

Accurate and fast localization of a predefined target region inside the patient is an important component of many image-guided therapy procedures. This problem is commonly solved by registration of intraoperative 2-D projection images to 3-D preoperative images. If the patient is not fixed during the intervention, the 2-D image acquisition is repeated several times during the procedure, and the registration problem can be cast instead as a 3-D tracking problem. To solve the 3-D problem, we propose in this paper to apply 2-D region tracking to first recover the components of the transformation that are in-plane to the projections. The 2-D motion estimates of all projections are backprojected into 3-D space, where they are then combined into a consistent estimate of the 3-D motion. We compare this method to intensity-based 2-D to 3-D registration and a combination of 2-D motion backprojection followed by a 2-D to 3-D registration stage. Using clinical data with a fiducial marker-based gold-standard transformation, we show that our method is capable of accurately tracking vertebral targets in 3-D from 2-D motion measured in X-ray projection images. Using a standard tracking algorithm (hyperplane tracking), tracking is achieved at video frame rates but fails relatively often (32% of all frames tracked with target registration error (TRE) better than 1.2 mm, 82% of all frames tracked with TRE better than 2.4 mm). With intensity-based 2-D to 2-D image registration using normalized mutual information (NMI) and pattern intensity (PI), accuracy and robustness are substantially improved. NMI tracked 82% of all frames in our data with TRE better than 1.2 mm and 96% of all frames with TRE better than 2.4 mm. This comes at the cost of a reduced frame rate, 1.7 s average processing time per frame and projection device. Results using PI were slightly more accurate, but required on average 5.4 s time per frame. These results are still substantially faster than 2-D to 3-D registration. We conclude that motion backprojection from 2-D motion tracking is an accurate and efficient method for tracking 3-D target motion, but tracking 2-D motion accurately and robustly remains a challenge.     

Tracking highly maneuverable targets with unknown behavior

Tracking of highly maneuvering targets with unknown behavior is a difficult problem in sequential state estimation. The performance of predictive-model-based Bayesian state estimators deteriorates quickly when their models are no longer accurate or their process noise is large. A data-driven approach to tracking, the segmenting track identifier (STI), is presented as an algorithm that operates well in environments where the measurement system is well understood but target motion is either or both highly unpredictable or poorly characterized. The STI achieves improved state estimates by the least-squares fitting of a motion model to a segment of data that has been partitioned from the total track such that it represents a single maneuver. Real-world STI tracking performance is demonstrated using sonar data collected from free-swimming fish, where the STI is shown to be effective at tracking highly maneuvering targets while relatively insensitive to its tuning parameters. Additionally, an extension of the STI to allow its use in the most common multiple target and cluttered environment data association frameworks is presented, and an STI-based joint probabilistic data association filter (STIJPDAF) is derived as a specific example. The STIJPDAF is shown by simulation to be effective at tracking a single fish in clutter and through empirical results from video data to be effective at simultaneously tracking multiple free-swimming fish.     

Alignment of 3-D optical coherence tomography scans to correct eye movement using a particle filtering

Eye movement artifacts occurring during 3-D optical coherence tomography (OCT) scanning is a well-recognized problem that may adversely affect image analysis and interpretation. A particle filtering algorithm is presented in this paper to correct motion in a 3-D dataset by considering eye movement as a target tracking problem in a dynamic system. The proposed particle filtering algorithm is an independent 3-D alignment approach, which does not rely on any reference image. 3-D OCT data is considered as a dynamic system, while the location of each A-scan is represented by the state space. A particle set is used to approximate the probability density of the state in the dynamic system. The state of the system is updated frame by frame to detect A-scan movement. The proposed method was applied on both simulated data for objective evaluation and experimental data for subjective evaluation. The sensitivity and specificity of the x-movement detection were 98.85% and 99.43%, respectively, in the simulated data. For the experimental data (74 3-D OCT images), all the images were improved after z-alignment, while 81.1% images were improved after x-alignment. The proposed algorithm is an efficient way to align 3-D OCT volume data and correct the eye movement without using references.     

Traffic accident prediction using 3-D model-based vehicle tracking

Intelligent visual surveillance for road vehicles is the key to developing autonomous intelligent traffic systems. Recently, traffic incident detection employing computer vision and image processing has attracted much attention. In this paper, a probabilistic model for predicting traffic accidents using three-dimensional (3-D) model-based vehicle tracking is proposed. Sample data including motion trajectories are first obtained by 3-D model-based vehicle tracking. A fuzzy self-organizing neural network algorithm is then applied to learn activity patterns from the sample trajectories. Finally, vehicle activity is predicted by locating and matching each partial trajectory with the learned activity patterns, and the occurrence probability of a traffic accident is determined. Experiments show the effectiveness of the proposed algorithms.     

3-D Kalman filter for image motion estimation

This paper presents a new three-dimensional (3-D) Markov model for motion vector fields. The three dimensions consist of the two space dimensions plus a scale dimension. We use a compound signal model to handle motion discontinuity in this 3-D Markov random field (MRF). For motion estimation, we use an extended Kalman filter as a pel-recursive estimator. Since a single observation can be sensitive to local image characteristics, especially when the model is not accurate, we employ windowed multiple observations at each pixel to increase accuracy. These multiple observations employ different weighting values for each observation, since the uncertainty in each observation is different. Finally, we compare this 3-D model with earlier proposed one-dimensional (1-D) (coarse-to-fine scale) and two-dimensional (2D) spatial compound models, in terms of motion estimation performance on a synthetic and a real image sequence.     

An efficient and accurate method for the relaxation of multiview registration error.

This paper presents a new method for the relaxation of multiview registration error. The multiview registration problem is represented using a graph. Each node and each edge in the graph represents a 3-D data set and a pairwise registration, respectively. Assuming that all the pairwise registration processes have converged to fine results, this paper shows that the multiview registration problem can be converted into a quadratic programming problem of Lie algebra parameters. The constraints are obtained from every cycle of the graph to eliminate the accumulation errors of global registration. A linear solution is proposed to distribute the accumulation error to proper positions in the graph, as specified by the quadratic model. Since the proposed method does not involve the original 3-D data, it has low time and space complexity. Additionally, the proposed method can be embedded into a trust-region algorithm and, thus, can correctly handle the nonlinear effects of large accumulation errors, while preserving the global convergence property to the first-order critical point. Experimental results confirm both the efficiency and the accuracy of the proposed method.     

Exploiting Motion Correlations in 3-D Articulated Human Motion Tracking

In 3-D articulated human motion tracking, the curse of dimensionality renders commonly-used particle-filter-based approaches inefficient. Also, noisy image measurements and imperfect feature extraction call for strong motion prior. We propose to learn the correlation between the right-side and the left-side human motion using partial least square (PLS) regression. The correlation effectively constrains the sampling of the proposal distribution to portions of the parameter space that correspond to plausible human motions. The learned correlation is then used as motion prior in designing a Rao–Blackwellized particle filter algorithm, RBPF-PLS, which estimates only one group of state variables using the Monte Carlo method, leaving the other group being exactly computed through an analytical filter that utilizes the learned motion correlation. We quantitatively assessed the accuracy of the proposed algorithm with challenging HumanEva-I/II data set. Experiments with comparison with both the annealed particle filter and the standard particle filter show that the proposed method achieves lower estimation error in processing challenging real-world data of 3-D human motion. In particular, the experiments demonstrate that the learned motion correlation model generalizes well to motions outside of the training set and is insensitive to the choice of the training subjects, suggesting the potential wide applicability of the method.      

A Novel Distributed Sensor Fusion Algorithm for RSSI-Based Location Estimation Using the Unscented Kalman Filter

We address the Bayesian sensor fusion approach for distributed location estimation in the wireless sensor network. Assume each sensor transmits local calculation of target position to a fusion center, which then generates under a Bayesian framework the final estimated trajectory. We study received signal strength indication-based approach using the unscented Kalman filter for each sensor to compute local estimation, and propose a novel distributed algorithm which combines the soft outputs sent from selected sensors and computes the approximated Bayesian estimates to the true position. Simulation results demonstrate that the proposed soft combining method can achieve similar tracking performance as the centralized data fusion approach. The computational cost of the proposed algorithm is less than the centralized method especially in large scale sensor networks. In addition, it is straightforward to incorporate the proposed soft combining strategy with other Bayesian filters for the general purpose of data fusion.

     

Tracking a dynamic set of feature points

We address the problems of tracking a set of feature points over a long sequence of monocular images as well as how to include and track new feature points detected in successive frames. Due to the 3-D movement of the camera, different parts of the images exhibit different image motion. Tracking discrete features can therefore be decomposed into several independent and local problems. Accordingly, we propose a localized feature tracking algorithm. The trajectory of each feature point is described by a 2-D kinematic model. Then to track a feature point, an interframe motion estimation scheme is designed to obtain the estimates of interframe motion parameters. Subsequently, using the estimates of motion parameters, corresponding points are identified to subpixel accuracy. Afterwards, the temporal information is processed to facilitate the tracking scheme. Since different feature points are tracked independently, the algorithm is able to handle the image motion arising from general 3-D camera movements. On the other hand, in addition to tracking feature points detected at the beginning, an efficient way to dynamically include new points extracted in subsequent frames is devised so that the information in a sequence is preserved. Experimental results for several image sequences are also reported.     

非线性高斯系统边缘分布多目标贝叶斯滤波器

为解决存在数据关联不确定、检测不确定和杂波情况下的多目标跟踪问题,提出了一种新的多目标贝叶斯滤波器.代替维持多目标状态的联合后验密度,所提出的贝叶斯滤波器联合传递各个目标状态的边缘分布和它们的存在概率.为了处理目标运动和传感器测量模型中的非线性,利用无迹变换技术提出了一种非线性高斯条件下边缘分布贝叶斯滤波器的近似实现算法.仿真实验结果表明,与PHD(Probability Hypothesis Density)滤波器相比,所提出的滤波器具有更好的多目标跟踪能力.     

Robust Neural-Network-Based Data Association and Multiple Model-Based Tracking of Multiple Point Targets

Data association and model selection are important factors for tracking multiple targets in a dense clutter environment without using a priori information about the target dynamic. We propose a neural-network-based tracking algorithm, incorporating a interacting multiple model and show that it is possible to track both maneuvering and nonmaneuvering targets simultaneously in the presence of dense clutter. Moreover, it can be used for real-time application. The proposed method overcomes the problem of data association by using the method of expectation maximization and Hopfield network to evaluate assignment weights. All validated observations are used to update the target state. In the proposed approach, a probability density function (pdf) of an observed data, given target state and observation association, is treated as a mixture pdf. This allows to combine the likelihood of an observation due to each model, and the association process is defined to incorporate an interacting multiple model, and consequently, it is possible to track any arbitrary trajectory     

Multiview vision-based human crowd localization for UAV fleet flight safety

This paper presents a centralized, vision-based method for robust, on-the-fly 3D localization and mapping of human crowds in large-scale outdoor environments, assuming their independent visual detection on the camera feed of multiple UAVs. The proposed method aims at enhancing vision-assisted human crowd avoidance, in line with common UAV safety regulations, since the resulting 3D crowd annotations may be employed by other algorithms for on-line mission/path replanning during deployment of a UAV fleet. Initially, 2D crowd heatmaps are assumed to be derived per video frame on-board each UAV separately, using deep neural human crowd detectors, which indicate the probability of each pixel depicting a human crowd. The UAV-mounted cameras are assumed to be covering the same large-scale outdoor area over time. The heatmaps of each time instance are transmitted to a central computer and back-projected onto the common 3D terrain/map of the navigation environment, utilizing the intrinsic and extrinsic camera parameters. The projected crowd heatmaps derived from the different drones/cameras are fused by exploiting a Bayesian filtering approach that favors newer crowd observations over older ones. Thus, during flight, an area is marked as crowded (therefore, a no-fly zone) if all, or most, individual UAV-mounted visual detectors have recently and confidently indicated crowd existence on it. In order to calculate prior probabilities for Bayesian fusion, the method also proposes and exploits a simple, but efficient image processing-based algorithm for identifying flat terrain areas (under the assumption that people do not gather on highly curved or inclined terrain), relying on a priori available ground elevation data for the mapped area. Evaluation on both synthetic and real-world multiview video sequences depicting human crowds in outdoor environments verifies the effectiveness of the proposed method.     

Predictive cardiac motion modeling and correction with partial least squares regression

Respiratory-induced cardiac deformation is a major problem for high-resolution cardiac imaging. This paper presents a new technique for predictive cardiac motion modeling and correction, which uses partial least squares regression to extract intrinsic relationships between three-dimensional (3-D) cardiac deformation due to respiration and multiple one-dimensional real-time measurable surface intensity traces at chest or abdomen. Despite the fact that these surface intensity traces can be strongly coupled with each other but poorly correlated with respiratory-induced cardiac deformation, we demonstrate how they can be used to accurately predict cardiac motion through the extraction of latent variables of both the input and output of the model. The proposed method allows cross-modality reconstruction of patient specific models for dense motion field prediction, which after initial modeling can be used for real-time prospective motion tracking or correction. Detailed numerical issues related to the technique are discussed and the effectiveness of the motion and deformation modeling is validated with 3-D magnetic resonance data sets acquired from ten asymptomatic subjects covering the entire respiratory range.     

基于最大置信度的多目标检测算法

针对混合运动模式下目标数量及目标运动速度范围等多项先验信息缺乏状况下的复杂航迹起始问题,提出一种基于最大置信度的多目标检测算法。该算法借鉴动态规划技术中的能量积累思想,并充分利用了传感器信号强度信息。在粗起始阶段利用扩展搜索算法生成候选航迹,并利用模型粗匹配的方法将候选航迹大致分为直线运动及曲线运动两种类型。在航迹确认阶段,采用基于信号强度信息的概率多假设跟踪算法,通过计算最优状态估计值获得量测点属于某一目标的最大置信度,并依据最大置信度确认目标量测。仿真实验结果表明,该方法实时性强,不仅能对多目标航迹准确起始,也可以有效避免概率多假设跟踪算法由于初值质量差而导致的错误跟踪现象。