Model-based temporal object verification using video

An approach to model-based dynamic object verification and identification using video is proposed. From image sequences containing the moving object, we compute its motion trajectory. Then we estimate its three-dimensional (3-D) pose at each time step. Pose estimation is formulated as a search problem, with the search space constrained by the motion trajectory information of the moving object and assumptions about the scene structure. A generalized Hausdorff (1962) metric, which is more robust to noise and allows a confidence interpretation, is suggested for the matching procedure used for pose estimation as well as the identification and verification problem. The pose evolution curves are used to assist in the acceptance or rejection of an object hypothesis. The models are acquired from real image sequences of the objects. Edge maps are extracted and used for matching. Results are presented for both infrared and optical sequences containing moving objects involved in complex motions     

无人机自主降落过程视觉定位方法研究

为了充分利用连续视觉图像中3维空间信息, 解决无人机自主降落过程中的定位问题, 在稠密3维点云法和光流法定位原理的基础上, 提出了基于同物不同时图像像空间的定位方法。以理论推算、图形注释等方式, 通过求解单个像素点和整个图像移动变化情况, 将连续帧图像的形变、量变信息分解为无人机和参照物的空间相对运动信息, 并结合已知的参照物运动参量, 推算了无人机飞行位姿信息, 完成了无人机基于光学视觉图像的空间定位方法研究。结果表明, 该研究为视觉系统在无人机降落回收过程中独立实现空间定位提供了一定的借鉴和参考。     

Computation and visualization of three-dimensional soft tissue motion in the orbit

This work presents a method to measure the soft tissue motion in three dimensions in the orbit during gaze. It has been shown that two-dimensional (2-D) quantification of soft tissue motion in the orbit is effective in the study of orbital anatomy and motion disorders. However, soft tissue motion is a three-dimensional (3-D) phenomenon and part of the kinematics is lost in any 2-D measurement. Therefore, T1-weighted magnetic resonance (MR) imaging volume sequences are acquired during gaze and soft tissue motion is quantified using a generalization of the Lucas and Kanade optical flow algorithm to three dimensions. New techniques have been developed for visualizing the 3-D flow field as a series of color-texture mapped 2-D slices or as a combination of volume rendering for display of the anatomy and scintillation rendering for the display of the motion field. We have studied the performance of the algorithm on four-dimensional volume sequences of synthetic motion, simulated motion of a static object imaged by MR, an MR-imaged rotating object and MR-imaged motion in the human orbit during gaze. The accuracy of the analysis is sufficient to characterize motion in the orbit and scintillation rendering is an effective visualization technique for 3-D motion in the orbit.     

视频图像中的车辆检测跟踪和分类

介绍了一种用固定的单摄像头拍摄交通图像,并从图像序列中检测、跟踪、分类车辆的方法。该方法大致可分为3部分：抽取背景图像和图像分割;基于针孔模型的摄像机定标,计算透视投影矩阵;利用区域特征进行匹配跟踪,建立目标链。恢复目标三维信息,采用模型匹配法对车型分类。实验证明方法是简单可行的。     

Two-dimensional mesh-based mosaic representation for manipulationof video objects with occlusion

We present a two-dimensional (2-D) mesh-based mosaic representation, consisting of an object mesh and a mosaic mesh for each frame and a final mosaic image, for video objects with mildly deformable motion in the presence of self and/or object-to-object (external) occlusion. Unlike classical mosaic representations where successive frames are registered using global motion models, we map the uncovered regions in the successive frames onto the mosaic reference frame using local affine models, i.e., those of the neighboring mesh patches. The proposed method to compute this mosaic representation is tightly coupled with an occlusion adaptive 2-D mesh tracking procedure, which consist of propagating the object mesh frame to frame, and updating of both object and mosaic meshes to optimize texture mapping from the mosaic to each instance of the object. The proposed representation has been applied to video object rendering and editing, including self transfiguration, synthetic transfiguration, and 2-D augmented reality in the presence of self and/or external occlusion. We also provide an algorithm to determine the minimum number of still views needed to reconstruct a replacement mosaic which is needed for synthetic transfiguration. Experimental results are provided to demonstrate both the 2-D mesh-based mosaic synthesis and two different video object editing applications on real video sequences.     

Background removal of multiview images by learning shape priors.

Image-based rendering has been successfully used to display 3-D objects for many applications. A well-known example is the object movie, which is an image-based 3-D object composed of a collection of 2-D images taken from many different viewpoints of a 3-D object. In order to integrate image-based 3-D objects into a chosen scene (e.g., a panorama), one has to meet a hard challenge--to efficiently and effectively remove the background from the foreground object. This problem is referred to as multiview images (MVIs) segmentation. Another task requires MVI segmentation is image-based 3-D reconstruction using multiview images. In this paper, we propose a new method for segmenting MVI, which integrates some useful algorithms, including the well-known graph-cut image segmentation and volumetric graph-cut. The main idea is to incorporate the shape prior into the image segmentation process. The shape prior introduced into every image of the MVI is extracted from the 3-D model reconstructed by using the volumetric graph cuts algorithm. Here, the constraint obtained from the discrete medial axis is adopted to improve the reconstruction algorithm. The proposed MVI segmentation process requires only a small amount of user intervention, which is to select a subset of acceptable segmentations of the MVI after the initial segmentation process. According to our experiments, the proposed method can provide not only good MVI segmentation, but also provide acceptable 3-D reconstructed models for certain less-demanding applications.     

New techniques for efficient sliding thin-slab volume visualization

High-resolution three-dimensional (3-D) volumetric images obtained by today's radiologic imaging scanners are rich in detailed diagnostic information. Despite the many visualization techniques available to assess such images, there remains information that is challenging to uncover, such as the location of small structures (e.g., mediastinal lymph nodes, narrowed-airway regions). Recently, sliding thin-slab (STS) visualization was proposed to improve the visualization of interior structures. These STS techniques sometimes depend on user opacity specifications or extra preprocessing, and other rendering approaches that use the general STS mechanism are conceivable. We introduce two techniques for STS volume visualization. The first, a depth (perspective) rendering process, produces an unobstructed, high-contrast 3-D view of the information within a thin volume of image data. Results are a function of relative planar locations. Thus, rendered views accurately depict the internal properties that were initially captured as position and intensity. The second method produces a gradient-like view of the intensity changes in a thin volume. Results can effectively detect the occurrence and location of dramatic tissue variations, often not visually recognized otherwise. Both STS techniques exploit the concept of temporal coherence to form sequences of consecutive slabs, using information from previously computed slabs. This permits efficient real-time computation on a general-purpose computer. Further, these techniques require no preprocessing, and results are not dependent on user knowledge. Results using 3-D computed tomography chest images show the computational efficiency and visual efficacy of the new STS techniques.     

Model evaluation and calibration for prospective respiratory motion correction in coronary MR angiography based on 3-D image registration

Image processing was used as a fundamental tool to derive motion information from magnetic resonance (MR) images, which was fed back into prospective respiratory motion correction during subsequent data acquisition to improve image quality in coronary MR angiography (CMRA) scans. This reduces motion artifacts in the images and, in addition, enables the usage of a broader gating window than commonly used today to increase the scan efficiency. The aim of the study reported in this paper was to find a suitable motion model to be used for respiratory motion correction in cardiac imaging and to develop a calibration procedure to adapt the motion model to the individual patient. At first, the performance of three motion models [one-dimensional translation in feet-head (FH) direction, three-dimensional (3-D) translation, and 3-D affine transformation] was tested in a small volunteer study. An elastic image registration algorithm was applied to 3-D MR images of the coronary vessels obtained at different respiratory levels. A strong intersubject variability was observed. The 3-D translation and affine transformation model were found to be superior over the conventional FH translation model used today. Furthermore, a new approach is presented, which utilizes a fast model-based image registration to extract motion information from time series of low-resolution 3-D MR images, which reflects the respiratory motion of the heart. The registration is based on a selectable global 3-D motion model (translation, rigid, or affine transformation). All 3-D MR images were registered with respect to end expiration. The resulting time series of model parameters were analyzed in combination with additionally acquired motion information from a diaphragmatic MR pencil-beam navigator to calibrate the respiratory motion model. To demonstrate the potential of a calibrated motion model for prospective motion correction in coronary imaging, the approach was tested in CMRA examinations in five volunteers.     

Pattern tracking and 3-D motion reconstruction of a rigid body from a 2-D image sequence

This paper presents an integrated method to identify an object pattern from an image, and track its movement over a sequence of images. The sequence of images comes from a single perspective video source, which is capturing data from a precalibrated scene. This information is used to reconstruct the scene in three-dimension (3-D) within a virtual environment where a user can interact and manipulate the system. The steps that are performed include the following: i) Identify an object pattern from a two-dimensional perspective video source. The user outlines the region of interest (ROI) in the initial frame; the procedure builds a refined mask of the dominant object within the ROI using the morphological watershed algorithm. ii) The object pattern is tracked between frames using object matching within the mask provided by the previous and next frame, computing the motion parameters. iii) The identified object pattern is matched with a library of shapes to identify a corresponding 3-D object. iv) A virtual environment is created to reconstruct the scene in 3-D using the 3-D object and the motion parameters. This method can be applied to real-life application problems, such as traffic management and material flow congestion analysis.     

Concurrent 3-D motion segmentation and 3-D interpretation of temporal sequences of monocular images.

The purpose of this study is to investigate a variational method for joint multiregion three-dimensional (3-D) motion segmentation and 3-D interpretation of temporal sequences of monocular images. Interpretation consists of dense recovery of 3-D structure and motion from the image sequence spatiotemporal variations due to short-range image motion. The method is direct insomuch as it does not require prior computation of image motion. It allows movement of both viewing system and multiple independently moving objects. The problem is formulated following a variational statement with a functional containing three terms. One term measures the conformity of the interpretation within each region of 3-D motion segmentation to the image sequence spatiotemporal variations. The second term is of regularization of depth. The assumption that environmental objects are rigid accounts automatically for the regularity of 3-D motion within each region of segmentation. The third and last term is for the regularity of segmentation boundaries. Minimization of the functional follows the corresponding Euler-Lagrange equations. This results in iterated concurrent computation of 3-D motion segmentation by curve evolution, depth by gradient descent, and 3-D motion by least squares within each region of segmentation. Curve evolution is implemented via level sets for topology independence and numerical stability. This algorithm and its implementation are verified on synthetic and real image sequences. Viewers presented with anaglyphs of stereoscopic images constructed from the algorithm's output reported a strong perception of depth.     

Cortex segmentation: a fast variational geometric approach

An automatic cortical gray matter segmentation from a three-dimensional (3-D) brain images [magnetic resonance (MR) or computed tomography] is a well known problem in medical image processing. In this paper, we first formulate it as a geometric variational problem for propagation of two coupled bounding surfaces. An efficient numerical scheme is then used to implement the geodesic active surface model. Experimental results of cortex segmentation on real 3-D MR data are provided.     

Estimation of 3-D left ventricular deformation from medical images using biomechanical models

The quantitative estimation of regional cardiac deformation from three-dimensional (3-D) image sequences has important clinical implications for the assessment of viability in the heart wall. We present here a generic methodology for estimating soft tissue deformation which integrates image-derived information with biomechanical models, and apply it to the problem of cardiac deformation estimation. The method is image modality independent. The images are segmented interactively and then initial correspondence is established using a shape-tracking approach. A dense motion field is then estimated using a transversely isotropic, linear-elastic model, which accounts for the muscle fiber directions in the left ventricle. The dense motion field is in turn used to calculate the deformation of the heart wall in terms of strain in cardiac specific directions. The strains obtained using this approach in open-chest dogs before and after coronary occlusion, exhibit a high correlation with strains produced in the same animals using implanted markers. Further, they show good agreement with previously published results in the literature. This proposed method provides quantitative regional 3-D estimates of heart deformation.     

Three-dimensional modeling from two-dimensional video

This paper presents the surface-based factorization method to recover three-dimensional (3-D) structure, i.e., the 3-D shape and 3-D motion, of a rigid object from a two-dimensional (2-D) video sequence. The main ingredients of our approach are as follows: 1) we describe the unknown shape of the 3-D rigid object by polynomial patches; 2) projections of these patches in the image plane move according to parametric 2-D motion models; 3) we recover the parameters describing the 3-D shape and 3-D motion from the 2-D motion parameters by factorizing a matrix that is rank 1 in a noiseless situation. Our method is simultaneously an extension and a simplification of the original factorization method of Tomasi and Kanade (1992). We track regions where the 2-D motion in the image plane is described by a single set of parameters, avoiding the need to track a large number of pointwise features, in general, a difficult task. Then our method estimates the parameters describing the 3-D structure by factoring a rank 1 matrix, not rank 3 as in Tomasi and Kanade. This allows the use of fast iterative algorithms to compute the 3-D structure that best fits the data. Experimental results with real-life video sequences illustrate the good performance of our approach     

Patch-Based Nonlocal Functional for Denoising Fluorescence Microscopy Image Sequences

We present a nonparametric regression method for denoising 3-D image sequences acquired via fluorescence microscopy. The proposed method exploits the redundancy of the 3-D+time information to improve the signal-to-noise ratio of images corrupted by Poisson-Gaussian noise. A variance stabilization transform is first applied to the image-data to remove the dependence between the mean and variance of intensity values. This preprocessing requires the knowledge of parameters related to the acquisition system, also estimated in our approach. In a second step, we propose an original statistical patch-based framework for noise reduction and preservation of space-time discontinuities. In our study, discontinuities are related to small moving spots with high velocity observed in fluorescence video-microscopy. The idea is to minimize an objective nonlocal energy functional involving spatio-temporal image patches. The minimizer has a simple form and is defined as the weighted average of input data taken in spatially-varying neighborhoods. The size of each neighborhood is optimized to improve the performance of the pointwise estimator. The performance of the algorithm (which requires no motion estimation) is then evaluated on both synthetic and real image sequences using qualitative and quantitative criteria.      

Multimodality Image Registration Using Spatial Procrustes Analysis and Modified Conditional Entropy

In this paper, we propose a new image registration technique using two kinds of information known as object shapes and voxel intensities. The proposed approach consists of two registration steps. First, an initial registration is carried out for two volume images by applying Procrustes analysis theory to the two sets of 3D feature points representing object shapes. During this first stage, a volume image is segmented by using a geometric deformable model. Then, 3D feature points are extracted from the boundary of a segmented object. We conduct an initial registration by applying Procrustes analysis theory with two sets of 3D feature points. Second, a fine registration is followed by using a new measure based on the entropy of conditional probabilities. Here, to achieve the final registration, we define a modified conditional entropy (MCE) computed from the joint histograms for voxel intensities of two given volume images. By using a two step registration method, we can improve the registration precision. To evaluate the performance of the proposed registration method, we conduct various experiments for our method as well as existing methods based on the mutual information (MI) and maximum likelihood (ML) criteria. We evaluate the precision of MI, ML and MCE-based measurements by comparing their registration traces obtained from magnetic resonance (MR) images and transformed computed tomography (CT) images with respect to x-translation and rotation. The experimental results show that our method has great potential for the registration of a variety of medical images.     

A comprehensive cardiac motion estimation framework using both untagged and 3-D tagged MR images based on nonrigid registration

In this paper, we present a novel technique based on nonrigid image registration for myocardial motion estimation using both untagged and 3-D tagged MR images. The novel aspect of our technique is its simultaneous usage of complementary information from both untagged and 3-D tagged MR images. To estimate the motion within the myocardium, we register a sequence of tagged and untagged MR images during the cardiac cycle to a set of reference tagged and untagged MR images at end-diastole. The similarity measure is spatially weighted to maximize the utility of information from both images. In addition, the proposed approach integrates a valve plane tracker and adaptive incompressibility into the framework. We have evaluated the proposed approach on 12 subjects. Our results show a clear improvement in terms of accuracy compared to approaches that use either 3-D tagged or untagged MR image information alone. The relative error compared to manually tracked landmarks is less than 15% throughout the cardiac cycle. Finally, we demonstrate the automatic analysis of cardiac function from the myocardial deformation fields.     

A direct multi-volume rendering method aiming at comparisons of 3-Dimages and models

The authors present a new method for direct volume rendering of multiple three-dimensional (3-D) functions using a density emitter model. The work is aimed at obtaining visual assessment of the results of a 3-D image registration algorithm which operates on anisotropic and non-segmented medical data. They first discuss the fundamentals associated with direct, simultaneous rendering of such datasets. Then, they recall the fuzzy classification and fuzzy surface rendering theory within the density emitter model terminology, and propose an extension of standard direct volume rendering that can handle the rendering of two or more 3-D functions; this consists of the definition of merging rules that are applied on emitter clouds. The included rendering applications are related on one hand, to volume-to-volume registration, and on the other hand, to surface-to-volume registration: the first case is concerned with global elastic registration of CT data, and the second one presents fitting of an implicit surface over a CT data subset. In these two medical imaging application cases, the rendering scheme offers a comprehensive appreciation of the relative position of structural information     

Second-order optimization of mutual information for real-time image registration

In this paper, we present a direct image registration approach that uses mutual information (MI) as a metric for alignment. The proposed approach is robust and gives an accurate estimation of a set of 2-D motion parameters in real time. MI is a measure of the quantity of information shared by signals. Although it has the ability to perform robust alignment with illumination changes, multimodality, and partial occlusions, few works have proposed MI-based applications related to spatiotemporal image registration or object tracking in image sequences because of some optimization problems, which we will explain. In this paper, we propose a new optimization method that is adapted to the MI cost function and gives a practical solution for real-time tracking. We show that by refining the computation of the Hessian matrix and using a specific optimization approach, the registration results are far more robust and accurate than the existing solutions, with the computation also being cheaper. A new approach is also proposed to speed up the computation of the derivatives and keep the same optimization efficiency. To validate the advantages of the proposed approach, several experiments are performed.     

Real-time multimodal medical image processing: a dynamicvolume-rendering application

A system for medical image processing has been proposed, which allows multimodal dynamic three-dimensional (3-D) visualization interactively and in real time. The system has been conceived to support medical specialists in the diagnosis of moving organs, such as the heart during the cardiac cycle, allowing them to compare information on perfusion/contraction match as a basis for diagnosis of important cardiovascular diseases. The 3-D volume rendering algorithm runs on a SIMD machine because of the great amount of data to be manipulated by always using the same operations. One of the features of the algorithm is the possibility to change, interactively, image processing and visualization parameters at any step, and to perform simple and effective image manipulations. Performance studies have demonstrated a very high global efficiency in practical situations by using typical data-volume dimensions. The system has been tested in the medical environment, by using magnetic resonance (MR) and single-photon emission-computed tomographic (SPECT) images     

Space-time image sequence analysis: object tunnels and occlusion volumes.

We address the issue of image sequence analysis jointly in space and time. While typical approaches to such an analysis consider two image frames at a time, we propose to perform this analysis jointly over multiple frames. We concentrate on spatiotemporal segmentation of image sequences and on analysis of occlusion effects therein. The segmentation process is three-dimensional (3-D); we search for a volume carved out by each moving object in the image sequence domain, or "object tunnel," a new space-time concept. We pose the problem in variational framework by using only motion information (no intensity edges). The resulting formulation can be viewed as volume competition, a 3-D generalization of region competition. We parameterize the unknown surface to be estimated, but rather than using an active-surface approach, we embed it into a higher dimensional function and apply the level-set methodology. We first develop simple models for the detection of moving objects over static background; no motion models are needed. Then, in order to improve segmentation accuracy, we incorporate motion models for objects and background. We further extend the method by including explicit models for occluded and newly exposed areas that lead to "occlusion volumes," another new space-time concept. Since, in this case, multiple volumes are sought, we apply a multiphase variant of the level-set method. We present various experimental results for synthetic and natural image sequences.