Three-dimensional motion estimation of objects for video coding

Three-dimensional (3-D) motion estimation is applied to the problem of motion compensation for video coding. We suppose that the video sequence consists of the perspective projections of a collection of rigid bodies which undergo a rototranslational motion. Motion compensation can be performed on the sequence once the shape of the objects and the motion parameters are determined. We show that the motion equations of a rigid body can be formulated as a nonlinear dynamic system whose state is represented by the motion parameters and by the scaled depths of the object feature points. An extended Kalman filter is used to estimate both the motion and the object shape parameters simultaneously. The inclusion of the shape parameters in the estimation procedure adds a set of constraints to the filter equations that appear to be essential for reliable motion estimation. Our experiments show that the proposed approach gives two advantages. First, the filter can give more reliable estimates in the presence of measurement noise in comparison with other motion estimators that separately compute motion and structure. Second, the filter can efficiently track abrupt motion changes. Moreover, the structure imposed by the model implies that the reconstructed motion is very natural as opposed to more common block-based schemes. Also, the parameterization of the model allows for a very efficient coding of the motion information     

3D Shape Inferencing and Modeling for Video Retrieval

We present a geometry-based indexing approach for the retrieval of video databases. It consists of two modules: 3D object shape inferencing from video data and geometric modeling from the reconstructed shape structure. A motion-based segmentation algorithm employing feature block tracking and principal component split is used for multi-moving-object motion classification and segmentation. After segmentation, feature blocks from each individual object are used to reconstruct its motion and structure through a factorization method. The estimated shape structure and motion parameters are used to generate the implicit polynomial model for the object. The video data is retrieved using the geometric structure of objects and their spatial relationship. We generalize the 2D string to 3D to compactly encode the spatial relationship of objects.     

Pattern tracking and 3-D motion reconstruction of a rigid body from a 2-D image sequence

This paper presents an integrated method to identify an object pattern from an image, and track its movement over a sequence of images. The sequence of images comes from a single perspective video source, which is capturing data from a precalibrated scene. This information is used to reconstruct the scene in three-dimension (3-D) within a virtual environment where a user can interact and manipulate the system. The steps that are performed include the following: i) Identify an object pattern from a two-dimensional perspective video source. The user outlines the region of interest (ROI) in the initial frame; the procedure builds a refined mask of the dominant object within the ROI using the morphological watershed algorithm. ii) The object pattern is tracked between frames using object matching within the mask provided by the previous and next frame, computing the motion parameters. iii) The identified object pattern is matched with a library of shapes to identify a corresponding 3-D object. iv) A virtual environment is created to reconstruct the scene in 3-D using the 3-D object and the motion parameters. This method can be applied to real-life application problems, such as traffic management and material flow congestion analysis.     

利用光流估计三维运动参数的线性算法

三维空间中物体运动参数可以用二维平面光流及图象平面上投影坐标求得。当物体表面结构小于物体与投影图象平面之间的距离时,算法是线性的。     

Estimation of depth fields suitable for video compression based on3-D structure and motion of objects

Intensity prediction along motion trajectories removes temporal redundancy considerably in video compression algorithms. In three-dimensional (3-D) object-based video coding, both 3-D motion and depth values are required for temporal prediction. The required 3-D motion parameters for each object are found by the correspondence-based E-matrix method. The estimation of the correspondences-two-dimensional (2-D) motion field-between the frames and segmentation of the scene into objects are achieved simultaneously by minimizing a Gibbs energy. The depth field is estimated by jointly minimizing a defined distortion and bit-rate criterion using the 3-D motion parameters. The resulting depth field is efficient in the rate-distortion sense. Bit-rate values corresponding to the lossless encoding of the resultant depth fields are obtained using predictive coding; prediction errors are encoded by a Lempel-Ziv algorithm. The results are satisfactory for real-life video scenes.     

Computation and visualization of three-dimensional soft tissue motion in the orbit

This work presents a method to measure the soft tissue motion in three dimensions in the orbit during gaze. It has been shown that two-dimensional (2-D) quantification of soft tissue motion in the orbit is effective in the study of orbital anatomy and motion disorders. However, soft tissue motion is a three-dimensional (3-D) phenomenon and part of the kinematics is lost in any 2-D measurement. Therefore, T1-weighted magnetic resonance (MR) imaging volume sequences are acquired during gaze and soft tissue motion is quantified using a generalization of the Lucas and Kanade optical flow algorithm to three dimensions. New techniques have been developed for visualizing the 3-D flow field as a series of color-texture mapped 2-D slices or as a combination of volume rendering for display of the anatomy and scintillation rendering for the display of the motion field. We have studied the performance of the algorithm on four-dimensional volume sequences of synthetic motion, simulated motion of a static object imaged by MR, an MR-imaged rotating object and MR-imaged motion in the human orbit during gaze. The accuracy of the analysis is sufficient to characterize motion in the orbit and scintillation rendering is an effective visualization technique for 3-D motion in the orbit.     

Two-dimensional mesh-based mosaic representation for manipulationof video objects with occlusion

We present a two-dimensional (2-D) mesh-based mosaic representation, consisting of an object mesh and a mosaic mesh for each frame and a final mosaic image, for video objects with mildly deformable motion in the presence of self and/or object-to-object (external) occlusion. Unlike classical mosaic representations where successive frames are registered using global motion models, we map the uncovered regions in the successive frames onto the mosaic reference frame using local affine models, i.e., those of the neighboring mesh patches. The proposed method to compute this mosaic representation is tightly coupled with an occlusion adaptive 2-D mesh tracking procedure, which consist of propagating the object mesh frame to frame, and updating of both object and mosaic meshes to optimize texture mapping from the mosaic to each instance of the object. The proposed representation has been applied to video object rendering and editing, including self transfiguration, synthetic transfiguration, and 2-D augmented reality in the presence of self and/or external occlusion. We also provide an algorithm to determine the minimum number of still views needed to reconstruct a replacement mosaic which is needed for synthetic transfiguration. Experimental results are provided to demonstrate both the 2-D mesh-based mosaic synthesis and two different video object editing applications on real video sequences.     

A factorization approach for cone-beam reconstruction on a circular short-scan

In this paper, we introduce a new algorithm for 3-D image reconstruction from cone-beam (CB) projections acquired along a partial circular scan. Our algorithm is based on a novel, exact factorization of the initial 3-D reconstruction problem into a set of independent 2-D inversion problems, each of which corresponds to finding the object density on one, single plane. Any such 2-D inversion problem is solved numerically using a projected steepest descent iteration scheme. We present a numerical evaluation of our factorization algorithm using computer-simulated CB data, without and with noise, of the FORBILD head phantom and of a disk phantom. First, we study quantitatively the impact of the reconstruction parameters on the algorithm performance. Next, we present reconstruction results for visual assessment of the achievable image quality and provide, for comparison, results obtained with two other state-of-the-art reconstruction algorithms for the circular short-scan.      

二维图像序列中刚性目标的准确定位方法

提出了一种准确定位二维图像序列中刚体目标的方法。方法使用粒子滤波,同时引入基于轮廓模型的匹配算法的基本思想,通过估算刚性目标的二维仿射运动参数,准确地计算其在每帧图像中的位置和姿态。利用基于距离变换的边缘匹配方法提高模型匹配效率和评价目标跟踪的精度。实验给出目标定位的计算结果并估算了定位精度,在图像序列中给出了计算所得的目标轮廓。实验表明．该方法的定位精度可达到亚pixel。     

Motion and structure from point correspondences with errorestimation: planar surfaces

The determination of motion and structure of a planar scene from monocular image sequences, is studied. A new and simpler linear algorithm, that gives closed-form solutions for motion and structure parameters using point correspondences between two images, assuming that the coplanar points undergo a rigid motion in 3-D space, is presented. A series of analytical results is established. From two perspective views of a planar scene which is undergoing a rigid motion, there are generally two (normalized) interpretations for motion parameters and the positions of the object plane. These two interpretations, one vertical and the other illusive, are both valid in the sense that they render the same pair of images. The authors identify all the special cases in which the number of interpretations is not two, and derive necessary and sufficient geometrical conditions for those special cases to occur. The approach to error estimation is based on the first-order perturbation. The estimated errors provide quantitative assessment for the accuracy of the solutions. They also indicate degenerate or nearly degenerate configurations in the presence of noise     

Three-dimensional modeling from endoscopic video using geometric constraints via feature positioning

The endoscope is a popular imaging modality used in many preevaluations and surgical treatments, and is also one of the essential tools in minimally invasive surgery. However, regular endoscopes provide only 2-D images. Even though stereoendoscopy systems can display 3-D images, the real anatomical structure of the observed lesion is unavailable and can only be judged by the surgeon's imagination. In this paper, we present a constraint-based factorization method for reconstructing 3-D structures registered to the patient, from 2-D endoscopic images. The proposed method incorporates the geometric constraints from the tracked surgical instrument into the traditional factorization method based on frame-to-frame feature motion on the endoscopically viewed scene. Experiments with real and synthetic data demonstrate good real-scale 3-D extraction, with greater accuracy than is available from traditional methods. The reconstruction process can also be accomplished in a few seconds, making it suitable for on-line surgical applications to provide surgeons with additional 3-D shape information, critical distance monitoring and warnings.     

Video representation with three-dimensional entities

Very low bit-rate coding requires new paradigms that go well beyond pixel- and frame-based video representations. We introduce a novel content-based video representation using tridimensional entities: textured object models and pose estimates. The multiproperty object models carry stochastic information about the shape and texture of each object present in the scene. The pose estimates define the position and orientation of the objects for each frame. This representation is compact. It provides alternative means for handling video by manipulating and compositing three-dimensional (3-D) entities. We call this representation tridimensional video compositing, or 3DVC for short. We present the 3DVC framework and describe the methods used to construct incrementally the object models and the pose estimates from unregistered noisy depth and texture measurements. We also describe a method for video frame reconstruction based on 3-D scene assembly, and discuss potential applications of 3DVC to video coding and content-based handling. 3DVC assumes that the objects in the scene are rigid and segmented. By assuming segmentation, we do not address the difficult questions of nonrigid segmentation and multiple object segmentation. In our experiments, segmentation is obtained via depth thresholding. It is important to notice that 3DVC is independent of the segmentation technique adopted. Experimental results with synthetic and real video sequences where compression ratios in the range of 1:150-1:2700 are achieved demonstrate the applicability of the proposed representation to very low bit-rate coding     

Segmentation and tracking of moving objects for content-based videocoding

To enable content-based functionalities in video coding, a decomposition of the scene into physical objects is required. Such objects are normally not characterised by homogeneous colour, intensity, or optical flow. Therefore, conventional techniques based on these low-level features cannot perform the desired segmentation. The authors address segmentation and tracking of moving objects and present a new video object plane (VOP) segmentation algorithm that extracts semantically meaningful objects. A morphological motion filter detects physical objects by identifying areas that are moving differently from the background. A new filter criterion is introduced that measures the deviation of the estimated local motion from the synthesised global motion. A two-dimensional binary model is derived for the object of interest and tracked throughout the sequence by a Hausdorff object tracker. To accommodate for rotations and changes in shape, the model is updated every frame by a two-stage method that accounts for rigid and non-rigid moving parts of the object. The binary model then guides the actual VOP extraction, whereby a novel boundary post-processor ensures high boundary accuracy. Experimental results demonstrate the performance of the proposed algorithm     

基于运动跟踪匹配技术的视频对象提取

本文提出一种自动分割VOP的技术。其方法是：先对初始帧使用形态运动滤波技术提取出初始运动对象的二值轮廓模型,并在后继帧中使用豪斯道夫对象跟踪器跟踪运动以对象模型;而为了适应对象的形状变化,本文使用活动轮廓模型（snake）技术对运动心合匹配;最后根据一系列精确的二值轮廓引导提取运动对象序列。实验结果表明,我们的算法可有效地提取视频对象平面。     

MRI artifact cancellation due to rigid motion in the imaging plane

A post-processing technique has been developed to suppress the magnetic resonance imaging (MRI) artifact arising from object planar rigid motion. In two-dimensional Fourier transform (2-DFT) MRI, rotational and translational motions of the target during magnetic resonance magnetic resonance (MR) scan respectively impose nonuniform sampling and a phase error an the collected MRI signal. The artifact correction method introduced considers the following three conditions: (1) for planar rigid motion with known parameters, a reconstruction algorithm based on bilinear interpolation and the super-position method is employed to remove the MRI artifact, (2) for planar rigid motion with known rotation angle and unknown translational motion (including an unknown rotation center), first, a super-position bilinear interpolation algorithm is used to eliminate artifact due to rotation about the center of the imaging plane, following which a phase correction algorithm is applied to reduce the remaining phase error of the MRI signal, and (3) to estimate unknown parameters of a rigid motion, a minimum energy method is proposed which utilizes the fact that planar rigid motion increases the measured energy of an ideal MR image outside the boundary of the imaging object; by using this property all unknown parameters of a typical rigid motion are accurately estimated in the presence of noise. To confirm the feasibility of employing the proposed method in a clinical setting, the technique was used to reduce unknown rigid motion artifact arising from the head movements of two volunteers.     

Temporal shape error concealment by global motion compensation with local refinement.

This paper presents an original temporal shape error concealment technique based on a combination of global and local motion compensation. For this technique, which is especially useful for object-based video applications in error-prone environments (e.g., mobile networks), it is assumed that the shape of the corrupted object at hand is in the form of a binary alpha plane and some of the shape data is missing due to channel errors. To conceal the corrupted shape, the decoder first assumes that a global motion model can describe the shape changes in consecutive time instants. This way, based on locally estimated global motion parameters, the decoder attempts to conceal the corrupted alpha plane by global motion compensating the shape data from the previous time instant. Afterwards, since a global motion model cannot perfectly describe all alpha plane changes, a local motion refinement is applied to improve the concealment in areas of the object with significant local motion.     

Three-dimensional surface reconstruction using optical flow formedical imaging

The recovery of a three-dimensional (3-D) model from a sequence of two-dimensional (2-D) images is very useful in medical image analysis. Image sequences obtained from the relative motion between the object and the camera or the scanner contain more 3-D information than a single image. Methods to visualize the computed tomograms can be divided into two approaches: the surface rendering approach and the volume rendering approach. In this paper, a new surface rendering method using optical flow is proposed. Optical flow is the apparent motion in the image plane produced by the projection of real 3-D motion onto the 2-D image. The 3-D motion of an object can be recovered from the optical-flow field using additional constraints. By extracting the surface information from 3-D motion, it is possible to obtain an accurate 3-D model of the object. Both synthetic and real image sequences have been used to illustrate the feasibility of the proposed method. The experimental results suggest that the proposed method is suitable for the reconstruction of 3-D models from ultrasound medical images as well as other computed tomograms     

Error concealment for shape in MPEG-4 object-based video coding.

In asynchronus transfer mode networks, cell loss or channel errors can cause data to be dropped in the channel. When digital images/videos are transmitted over these networks, one must be able to reconstruct the missing data so that the impact of the errors is minimized. In this paper, we present an error-concealment technique for shape in MPEG-4 object-based video coding. This method, which is based on using global motion estimation and compensation techniques for boundary recovery, consists of three steps: (1) boundary extraction from shape; (2) boundary patching using global motion compensation; and (3) boundary filling to reconstruct the shape of the damaged video object planes. Global motion parameters are inserted as part of the USER_DATA field in the compressed stream and are utilized in reconstructing the damaged boundaries of compressed video object planes.