Moving object detection and tracking from video captured by moving camera

This paper presents an effective method for the detection and tracking of multiple moving objects from a video sequence captured by a moving camera without additional sensors. Moving object detection is relatively difficult for video captured by a moving camera, since camera motion and object motion are mixed. In the proposed method, the feature points in the frames are found and then classified as belonging to foreground or background features. Next, moving object regions are obtained using an integration scheme based on foreground feature points and foreground regions, which are obtained using an image difference scheme. Then, a compensation scheme based on the motion history of the continuous motion contours obtained from three consecutive frames is applied to increase the regions of moving objects. Moving objects are detected using a refinement scheme and a minimum bounding box. Finally, moving object tracking is achieved using a Kalman filter based on the center of gravity of a moving object region in the minimum bounding box. Experimental results show that the proposed method has good performance.     

3-D model-based frame interpolation for distributed video coding of static scenes.

This paper addresses the problem of side information extraction for distributed coding of videos captured by a camera moving in a 3-D static environment. Examples of targeted applications are augmented reality, remote-controlled robots operating in hazardous environments, or remote exploration by drones. It explores the benefits of the structure-from-motion paradigm for distributed coding of this type of video content. Two interpolation methods constrained by the scene geometry, based either on block matching along epipolar lines or on 3-D mesh fitting, are first developed. These techniques are based on a robust algorithm for sub-pel matching of feature points, which leads to semi-dense correspondences between key frames. However, their rate-distortion (RD) performances are limited by misalignments between the side information and the actual Wyner-Ziv (WZ) frames due to the assumption of linear motion between key frames. To cope with this problem, two feature point tracking techniques are introduced, which recover the camera parameters of the WZ frames. A first technique, in which the frames remain encoded separately, performs tracking at the decoder and leads to significant RD performance gains. A second technique further improves the RD performances by allowing a limited tracking at the encoder. As an additional benefit, statistics on tracks allow the encoder to adapt the key frame frequency to the video motion content.     

A comparison framework for breathing motion estimation methods from 4-D imaging

Motion estimation is an important issue in radiation therapy of moving organs. In particular, motion estimates from 4-D imaging can be used to compute the distribution of an absorbed dose during the therapeutic irradiation. We propose a strategy and criteria incorporating spatiotemporal information to evaluate the accuracy of model-based methods capturing breathing motion from 4-D CT images. This evaluation relies on the identification and tracking of landmarks on the 4-D CT images by medical experts. Three different experts selected more than 500 landmarks within 4-D CT images of lungs for three patients. Landmark tracking was performed at four instants of the expiration phase. Two metrics are proposed to evaluate the tracking performance of motion-estimation models. The first metric cumulates over the four instants the errors on landmark location. The second metric integrates the error over a time interval according to an a priori breathing model for the landmark spatiotemporal trajectory. This latter metric better takes into account the dynamics of the motion. A second aim of this paper is to estimate the impact of considering several phases of the respiratory cycle as compared to using only the extreme phases (end-inspiration and end-expiration). The accuracy of three motion estimation models (two image registration-based methods and a biomechanical method) is compared through the proposed metrics and statistical tools. This paper points out the interest of taking into account more frames for reliably tracking the respiratory motion.     

Video orbits of the projective group a simple approach tofeatureless estimation of parameters

We present direct featureless methods for estimating the eight parameters of an "exact" projective (homographic) coordinate transformation to register pairs of images, together with the application of seamlessly combining a plurality of images of the same scene, resulting in a single image (or new image sequence) of greater resolution or spatial extent. The approach is "exact" for two cases of static scenes: (1) images taken from the same location of an arbitrary three-dimensional (3-D) scene, with a camera that is free to pan, tilt, rotate about its optical axis, and zoom, or (2) images of a flat scene taken from arbitrary locations. The featureless projective approach generalizes interframe camera motion estimation methods that have previously used a camera model (which lacks the degrees of freedom to "exactly" characterize such phenomena as camera pan and tilt) and/or which have relied upon finding points of correspondence between the image frames. The featureless projective approach, which operates directly on the image pixels, is shown to be superior in accuracy and the ability to enhance the resolution. The proposed methods work well on image data collected from both good-quality and poor-quality video under a wide variety of conditions (sunny, cloudy, day, night). These new fully automatic methods are also shown to be robust to deviations from the assumptions of static scene and no parallax.     

Reversible interframe compression of medical images: a comparison of decorrelation methods

The authors investigate whether data representing medical image sequences can be compressed more efficiently by taking into account the temporal correlation between the sequence frames. The standard of comparison is intraframe HINT, the best-known reversible decorrelation method for 2-D images. In interframe decorrelation, a distinction is made between extrapolation- and interpolation-based methods, and methods based on local motion estimation, block motion estimation, and unregistered decorrelation. These distinctions give six classes of interframe decorrelation methods, all of which are described. The methods are evaluated by applying them to sequences of coronary X-ray angiograms, ventricle angiograms, and liver scintigrams, as well as to a (nonmedical) videoconferencing image sequence. For the medical image sequences: (1) interpolation-based methods are superior to extrapolation-based methods; (2) estimation of interframe motion is not advantageous for image compression; (3) interframe compression yields entropies comparable to intraframe HINT at higher computational costs; and (4) two methods, unregistered extrapolation and interpolation, are nonetheless possibly interesting alternatives to intraframe HINT.     

Markerless real-time 3-D target region tracking by motion backprojection from projection images

Accurate and fast localization of a predefined target region inside the patient is an important component of many image-guided therapy procedures. This problem is commonly solved by registration of intraoperative 2-D projection images to 3-D preoperative images. If the patient is not fixed during the intervention, the 2-D image acquisition is repeated several times during the procedure, and the registration problem can be cast instead as a 3-D tracking problem. To solve the 3-D problem, we propose in this paper to apply 2-D region tracking to first recover the components of the transformation that are in-plane to the projections. The 2-D motion estimates of all projections are backprojected into 3-D space, where they are then combined into a consistent estimate of the 3-D motion. We compare this method to intensity-based 2-D to 3-D registration and a combination of 2-D motion backprojection followed by a 2-D to 3-D registration stage. Using clinical data with a fiducial marker-based gold-standard transformation, we show that our method is capable of accurately tracking vertebral targets in 3-D from 2-D motion measured in X-ray projection images. Using a standard tracking algorithm (hyperplane tracking), tracking is achieved at video frame rates but fails relatively often (32% of all frames tracked with target registration error (TRE) better than 1.2 mm, 82% of all frames tracked with TRE better than 2.4 mm). With intensity-based 2-D to 2-D image registration using normalized mutual information (NMI) and pattern intensity (PI), accuracy and robustness are substantially improved. NMI tracked 82% of all frames in our data with TRE better than 1.2 mm and 96% of all frames with TRE better than 2.4 mm. This comes at the cost of a reduced frame rate, 1.7 s average processing time per frame and projection device. Results using PI were slightly more accurate, but required on average 5.4 s time per frame. These results are still substantially faster than 2-D to 3-D registration. We conclude that motion backprojection from 2-D motion tracking is an accurate and efficient method for tracking 3-D target motion, but tracking 2-D motion accurately and robustly remains a challenge.     

一种基于特征点的快速跟踪算法

目标跟踪技术是视频检测技术中一个十分重要的组成部分,为此,提出一种基于特征点的快速跟踪算法。该方法避免了困难的目标分割过程。采用两次帧差共同确定角点选择区域,利用Moravec算法提取合适角点;采用一种特别设计的包含不平滑区域的结构化模板获取更好的匹配点;利用预测点缩小搜索范围,降低计算复杂度和时间复杂度。实验证明该算法能够快速实现目标的实时跟踪,跟踪准确度高,对不同的场景都具有良好的鲁棒性。     

Image sequence stabilisation based on DFT filtering

An image sequence stabilisation system based on DFT filtering of absolute frame displacements to compensate for undesired fluctuations in the sequence by shifting image frames into correct positions is reported. The system compensates for undesired jitter, while preserving desired global camera motions. Robustness is introduced to interframe motion estimation by averaging motion vectors detected in the phase correlation surface     

Multicamera Tracking of Articulated Human Motion Using Shape and Motion Cues

We present a completely automatic algorithm for initializing and tracking the articulated motion of humans using image sequences obtained from multiple cameras. A detailed articulated human body model composed of sixteen rigid segments that allows both translation and rotation at joints is used. Voxel data of the subject obtained from the images is segmented into the different articulated chains using Laplacian Eigenmaps. The segmented chains are registered in a subset of the frames using a single-frame registration technique and subsequently used to initialize the pose in the sequence. A temporal registration method is proposed to identify the partially segmented or unregistered articulated chains in the remaining frames in the sequence. The proposed tracker uses motion cues such as pixel displacement as well as 2-D and 3-D shape cues such as silhouettes, motion residue, and skeleton curves. The tracking algorithm consists of a predictor that uses motion cues and a corrector that uses shape cues. The use of complementary cues in the tracking alleviates the twin problems of drift and convergence to local minima. The use of multiple cameras also allows us to deal with the problems due to self-occlusion and kinematic singularity. We present tracking results on sequences with different kinds of motion to illustrate the effectiveness of our approach. The pose of the subject is correctly tracked for the duration of the sequence as can be verified by inspection.      

针对间断纹理环境中的图像特征追踪和匹配算法研究

针对同时定位与地图构建(SLAM)中的特征匹配关键环节,提出一种融合特征点和特征区域的图像追踪与匹配算法,以解决交替出现纹理丰富和纹理缺失的间断纹理环境中图像特征易丢失、误匹配率较高的问题。首先,利用ORB算法和半稠密直接法分别对图像提取特征点和特征区域。其次,使用渐进一致采样法(PROSAC)剔除ORB算法的误匹配特征点,并计算特征点的正确匹配率。最后,针对纹理缺失环境中特征点丢失严重的问题,以特征点的正确匹配率作为判断依据,对低匹配率图像,则基于特征区域使用半稠密直接法实现图像的追踪,同时对追踪结果进行非线性优化,提高了特征区域追踪的准确性和稳定性。实验结果表明,该算法适用于间断纹理环境,在纹理丰富和纹理缺失条件下均可提高图像匹配的准确率。     

Shape-based tracking of left ventricular wall motion

An approach for tracking and quantifying the nonrigid, nonuniform motion of the left ventricular (LV) endocardial wall from two-dimensional (2-D) cardiac image sequences, on a point-by-point basis over the entire cardiac cycle, is presented. Given a set of boundaries, motion computation involves first matching local segments on one contour to segments on the next contour in the sequence using a shape-based strategy. Results from the match process are incorporated with a smoothness term into an optimization functional. The global minimum of this functional is found, resulting in a smooth flow field that is consistent with the match data. The computation is performed for all pairs of frames in the temporal sequence and equally sampled points on one contour are tracked throughout the sequence, resulting in a composite flow field over the entire sequence. Two perspectives on characterizing the optimization functional are presented which result in a tradeoff resolved by the confidence in the initial boundary segmentation. Experimental results for contours derived from diagnostic image sequences of three different imaging modalities are presented. A comparison of trajectory estimates with trajectories of gold-standard markers implanted in the LV wall are presented for validation. The results of this comparison confirm that although cardiac motion is a three-dimensional (3-D) problem, two-dimensional (2-D) analysis provides a rich testing ground for algorithm development     

Motion Compensated Subband Video Coding with Arbitrarily Shaped Region Adaptivity

The performance of Motion Compensated Discrete Cosine Transform (MC‐DCT) video coding is improved by using the region adaptive subband image coding [18]. On the assumption that the video is acquired from the camera on a moving platform and the distance between the camera and the scene is large enough, both the motion of camera and the motion of moving objects in a frame are compensated. For the compensation of camera motion, a feature matching algorithm is employed. Several feature points extracted using a Sobel operator are used to compensate the camera motion of translation, rotation, and zoom. The illumination change between frames is also compensated. Motion compensated frame differences are divided into three regions called stationary background, moving objects, and newly emerging areas each of which is arbitrarily shaped. Different quantizers are used for different regions. Compared to the conventional MC‐DCT video coding using block matching algorithm, our video coding scheme shows about 1.0‐dB improvements on average for the experimental video samples.     

Active mesh-a feature seeking and tracking image sequencerepresentation scheme

This paper introduces a representation scheme for image sequences using nonuniform samples embedded in a deformable mesh structure. It describes a sequence by nodal positions and colors in a starting frame, followed by nodal displacements in the following frames. The nodal points in the mesh are more densely distributed in regions containing interesting features such as edges and corners; and are dynamically updated to follow the same features in successive frames. They are determined automatically by maximizing feature (e.g., gradient) magnitudes at nodal points, while minimizing interpolation errors within individual elements, and matching errors between corresponding elements. In order to avoid the mesh elements becoming overly deformed, a penalty term is also incorporated, which measures the irregularity of the mesh structure. The notions of shape functions and master elements commonly used in the finite element method have been applied to simplify the numerical calculation of the energy functions and their gradients. The proposed representation is motivated by the active contour or snake model proposed by Kass, Witkin, and Terzopoulos (1988). The current representation retains the salient merit of the original model as a feature tracker based on local and collective information, while facilitating more accurate image interpolation and prediction. Our computer simulations have shown that the proposed scheme can successfully track facial feature movements in head-and-shoulder type of sequences, and more generally, interframe changes that can be modeled as elastic deformation. The treatment for the starting frame also constitutes an efficient representation of arbitrary still images.     

Affine-structure-based facial image encoding

A real-time algorithm for affine-structure-based video compression for facial images is presented. The face undergoing motion is segmented and triangulated to yield a set of control points. The set of control points generated by triangulation are tracked across a few frames using an intensity-based correlation technique. For accurate motion and structure estimation a Kalman-filter-based algorithm is used to track features on the facial image. The structure information of the control points is transmitted only during the bootstrapping stage. After that only the motion information is transmitted to the decoder. This reduces the number of motion parameters associated with control points in each frame. The local motion of the eyes and lips is captured using local 2-D affine transformations. For real time implementation a quad-tree based search technique is adopted to solve local correlation. Any remaining reconstruction error is accounted for using predictive encoding. Results on real image sequences demonstrate the applicability of the method     

Mean-Shift Object Tracking with Discrete and Real AdaBoost Techniques

An online mean-shift object tracking algorithm, which consists of a learning stage and an estimation stage, is proposed in this work. The learning stage selects the features for tracking, and the estimation stage composes a likelihood image and applies the mean shift algorithm to it to track an object. The tracking performance depends on the quality of the likelihood image. We propose two schemes to generate and integrate likelihood images: one based on the discrete AdaBoost (DAB) and the other based on the real AdaBoost (RAB). The DAB scheme uses tuned feature values, whereas RAB estimates class probabilities, to select the features and generate the likelihood images. Experiment results show that the proposed algorithm provides more accurate and reliable tracking results than the conventional mean shift tracking algorithms.     

基于双目视觉模型的运动参数测量

通过建立双目测量的数学模型,由标志点在双摄像机成像平面上的坐标求得运动物体的运动参数。在不考虑摄像机内外参数标定误差的情况下,研究了标志点像元的测量精度对运动参数测量的影响。通过分析图像分割求取标志点形心引起的误差,利用窗口中心迭代算法提高标志点形心测量的精度,从而提高了运动参数的测量精度。由空间飞行器自动交会的仿真试验验证了算法的有效性。     

Three-dimensional motion tracking of coronary arteries in biplane cineangiograms

A three-dimensional (3-D) method for tracking the coronary arteries through a temporal sequence of biplane X-ray angiography images is presented. A 3-D centerline model of the coronary vasculature is reconstructed from a biplane image pair at one time frame, and its motion is tracked using a coarse-to-fine hierarchy of motion models. Three-dimensional constraints on the length of the arteries and on the spatial regularity of the motion field are used to overcome limitations of classical two-dimensional vessel tracking methods, such as tracking vessels through projective occlusions. This algorithm was clinically validated in five patients by tracking the motion of the left coronary tree over one cardiac cycle. The root mean square reprojection errors were found to be submillimeter in 93% (54/58) of the image pairs. The performance of the tracking algorithm was quantified in three dimensions using a deforming vascular phantom. RMS 3-D distance errors were computed between centerline models tracked in the X-ray images and gold-standard centerline models of the phantom generated from a gated 3-D magnetic resonance image acquisition. The mean error was 0.69 (+/- 0.06) mm over eight temporal phases and four different biplane orientations.     

Pattern tracking and 3-D motion reconstruction of a rigid body from a 2-D image sequence

This paper presents an integrated method to identify an object pattern from an image, and track its movement over a sequence of images. The sequence of images comes from a single perspective video source, which is capturing data from a precalibrated scene. This information is used to reconstruct the scene in three-dimension (3-D) within a virtual environment where a user can interact and manipulate the system. The steps that are performed include the following: i) Identify an object pattern from a two-dimensional perspective video source. The user outlines the region of interest (ROI) in the initial frame; the procedure builds a refined mask of the dominant object within the ROI using the morphological watershed algorithm. ii) The object pattern is tracked between frames using object matching within the mask provided by the previous and next frame, computing the motion parameters. iii) The identified object pattern is matched with a library of shapes to identify a corresponding 3-D object. iv) A virtual environment is created to reconstruct the scene in 3-D using the 3-D object and the motion parameters. This method can be applied to real-life application problems, such as traffic management and material flow congestion analysis.     

Video compression using mosaic representations

We describe a technique for video compression based on a mosaic image representation obtained by aligning all frames of a video sequence, giving a panoramic view of the scene. We describe two types of mosaics, static and dynamic, which are suited for storage and transmission applications, respectively. In each case, the mosaic construction process aligns the images using a global parametric motion transformation, usually canceling the effect of camera motion on the dominant portion of the scene. The residual motions that are not compensated by the parametric motion are then analyzed for their significance and coded. The mosaic representation exploits large scale spatial and temporal correlations in image sequences. In many applications where there is significant camera motion (e.g., remote surveillance), it performs substantially better than traditional interframe compression methods and offers the potential for very low bit-rate transmission. In storage applications, such as digital libraries and video editing environments, it has the additional benefit of enabling direct access and retrieval of single frames at a time.     

矿井车载摄像系统中电子稳像算法的研究

为克服矿井车载摄像系统在视频采集过程中的图像抖动问题,提出了一种基于MIC-Harris角点特征的自适应运动补偿电子稳像算法。首先,提出MIC-Harris角点检测算法和由粗到精的匹配策略,实现序列中特征点的快速精确提取。然后根据矿井移动车载图像运动特点,构建运动参数估计模型,运用最小二乘法实现全局运动矢量的求解。最后,提出自适应相邻帧补偿方法,结合Kalman滤波实现补偿矢量的计算,在滤除高频随机抖动同时保留了车载摄像机低频主动运动,并克服了连续视频序列长时间稳像处理的帧间累积误差问题。实验仿真结果表明：本文提出的算法不仅融合了MIC运算速度快和Harris准确度高的优点,而且能够实现矿井车载摄像系统的长时间连续稳像处理,具有较好的稳像效果。