Review and Preview: Disocclusion by Inpainting for Image-Based Rendering

Image-based rendering takes as input multiple images of an object and generates photorealistic images from novel viewpoints. This approach avoids explicitly modeling scenes by replacing the modeling phase with an object reconstruction phase. Reconstruction is achieved in two possible ways: recovering 3-D point locations using multiview stereo techniques, or reasoning about consistency of each voxel in a discretized object volume space. The most challenging problem for image-based reconstruction is the presence of occlusions. Occlusions make reconstruction ambiguous for object parts not visible in any input image. These parts must be reconstructed in a visually acceptable way. This paper both reviews image inpainting and argues that inpainting can provide not only attractive reconstruction but also a framework for increasing the accuracy of depth recovery. Digital image inpainting refers to any methods that fill-in holes of arbitrary topology in images so that they seem to be a part of the original image. Available methods are broadly classified as structural inpainting or textural inpainting. Structural inpainting reconstructs using prior assumptions and boundary conditions, while textural inpainting considers only the available data from texture exemplars or other templates. Of particular particular interest is research on structural inpainting applied to 3-D models, emphasizing its effectiveness for disocclusion.     

Hierarchical scale-based multiobject recognition of 3-D anatomical structures

Segmentation of anatomical structures from medical images is a challenging problem, which depends on the accurate recognition (localization) of anatomical structures prior to delineation. This study generalizes anatomy segmentation problem via attacking two major challenges: 1) automatically locating anatomical structures without doing search or optimization, and 2) automatically delineating the anatomical structures based on the located model assembly. For 1), we propose intensity weighted ball-scale object extraction concept to build a hierarchical transfer function from image space to object (shape) space such that anatomical structures in 3-D medical images can be recognized without the need to perform search or optimization. For 2), we integrate the graph-cut (GC) segmentation algorithm with prior shape model. This integrated segmentation framework is evaluated on clinical 3-D images consisting of a set of 20 abdominal CT scans. In addition, we use a set of 11 foot MR images to test the generalizability of our method to the different imaging modalities as well as robustness and accuracy of the proposed methodology. Since MR image intensities do not possess a tissue specific numeric meaning, we also explore the effects of intensity nonstandardness on anatomical object recognition. Experimental results indicate that: 1) effective recognition can make the delineation more accurate; 2) incorporating a large number of anatomical structures via a model assembly in the shape model improves the recognition and delineation accuracy dramatically; 3) ball-scale yields useful information about the relationship between the objects and the image; 4) intensity variation among scenes in an ensemble degrades object recognition performance.     

Multiview deblurring for 3-D images from light-sheet-based fluorescence microscopy

We propose an algorithm for 3-D multiview deblurring using spatially variant point spread functions (PSFs). The algorithm is applied to multiview reconstruction of volumetric microscopy images. It includes registration and estimation of the PSFs using irregularly placed point markers (beads). We formulate multiview deblurring as an energy minimization problem subject to L1-regularization. Optimization is based on the regularized Lucy-Richardson algorithm, which we extend to deal with our more general model. The model parameters are chosen in a profound way by optimizing them on a realistic training set. We quantitatively and qualitatively compare with existing methods and show that our method provides better signal-to-noise ratio and increases the resolution of the reconstructed images.     

Volumetric object reconstruction using the 3D-MRF model-basedsegmentation [magnetic resonance imaging]

A number of segmentation algorithms have been developed, but those algorithms are not effective on volume reconstruction because they are limited to operating only on two-dimensional (2-D) images. Here, the authors propose the volumetric object reconstruction method using the three-dimensional Markov random field (3D-MRF) model-based segmentation. The 3D-MRF model is known to be one of the most efficient ways to model spatial contextual information. The method is compared with the 2-D region growing scheme under three types of interpolation. The results show that the proposed method is better in terms of image quality than the other methods     

Video representation with three-dimensional entities

Very low bit-rate coding requires new paradigms that go well beyond pixel- and frame-based video representations. We introduce a novel content-based video representation using tridimensional entities: textured object models and pose estimates. The multiproperty object models carry stochastic information about the shape and texture of each object present in the scene. The pose estimates define the position and orientation of the objects for each frame. This representation is compact. It provides alternative means for handling video by manipulating and compositing three-dimensional (3-D) entities. We call this representation tridimensional video compositing, or 3DVC for short. We present the 3DVC framework and describe the methods used to construct incrementally the object models and the pose estimates from unregistered noisy depth and texture measurements. We also describe a method for video frame reconstruction based on 3-D scene assembly, and discuss potential applications of 3DVC to video coding and content-based handling. 3DVC assumes that the objects in the scene are rigid and segmented. By assuming segmentation, we do not address the difficult questions of nonrigid segmentation and multiple object segmentation. In our experiments, segmentation is obtained via depth thresholding. It is important to notice that 3DVC is independent of the segmentation technique adopted. Experimental results with synthetic and real video sequences where compression ratios in the range of 1:150-1:2700 are achieved demonstrate the applicability of the proposed representation to very low bit-rate coding     

VOIR: a volumetric image reconstruction algorithm based on Fouriertechniques for inversion of the 3-D Radon transform

A novel volumetric image reconstruction algorithm known as VOIR is presented for inversion of the 3-D Radon transform or its radial derivative. The algorithm is a direct implementation of the projection slice theorem for plane integrals. It generalizes one of the most successful methods in 2-D Fourier image reconstruction involving concentric-square rasters to 3-D; in VOIR, the spectral data, which is calculated by fast Fourier techniques, lie on concentric cubes and are interpolated by a bilinear method on the sides of these concentric cubes. The algorithm has great computational advantages over filtered-backprojection algorithms; for images of side dimension N, the numerical complexity of VOIR is O(N(3) log N) instead of O(N (4)) for backprojection techniques. An evaluation of the image processing performance is reported by comparison of reconstructed images from simulated cone-beam scans of a contrast and resolution test object. The image processing performance is also characterized by an analysis of the edge response from the reconstructed images.     

Tracking monotonically advancing boundaries in image sequences using graph cuts and recursive kernel shape priors

We introduce a probabilistic computer vision technique to track monotonically advancing boundaries of objects within image sequences. Our method incorporates a novel technique for including statistical prior shape information into graph-cut based segmentation, with the aid of a majorization-minimization algorithm. Extension of segmentation from single images to image sequences then follows naturally using sequential Bayesian estimation. Our methodology is applied to two unrelated sets of real biomedical imaging data, and a set of synthetic images. Our results are shown to be superior to manual segmentation.     

New approach to extract phase feature of 3-D objects by wavelength-scanning digital holography

Phaseinformationisoneofkeyfeaturesfor3Dob jectsespeciallyforalmosttransparentobject,suchas somebiologicaltissues.Thus,phasefeaturesaremost widelyappliedinrecognization,opticalmetrologyandsoon.Somemethodshavebeenproposedtoobtainthe phaseinformation[14]ofa3…     

Towards 3-D scene reconstruction from broadcast video

Three-dimensional (3-D) scene reconstruction from broadcast video is a challenging problem with many potential applications, such as 3-D TV, free-view TV, augmented reality or three-dimensionalization of two-dimensional (2-D) media archives. In this paper, a flexible and effective system capable of efficiently reconstructing 3-D scenes from broadcast video is proposed, with the assumption that there is relative motion between camera and scene/objects. The system requires no a priori information and input, other than the video sequence itself, and capable of estimating the internal and external camera parameters and performing a 3-D motion-based segmentation, as well as computing a dense depth field. The system also serves as a showcase to present some novel approaches for moving object segmentation, sparse and dense reconstruction problems. According to the simulations for both synthetic and real data, the system achieves a promising performance for typical TV content, indicating that it is a significant step towards the 3-D reconstruction of scenes from broadcast video.     

基于傅里叶变换去隔行图像的动态3维面形测量

为了解决动态面形测量中隔行扫描CCD相机记录动态物体表面变形条纹图像存在缺陷的问题,提出了傅里叶变换去隔行算法,即把隔行扫描CCD获取动态物体的错位模糊帧图像分成两个单场图像,分别对每一单场图像进行傅里叶变换去隔行处理,再利用条纹分析法重建对应时刻的3维面形。理论分析得出单场傅里叶变换去隔行图像与对应的准确满帧图像相同的结论。结果表明,该方法可以很好地恢复条纹和重建物体,且简单实用,可用于基于空间相位检测、相位测量轮廓术、傅里叶变换轮廓术等条纹分析方法的动态物体3维测量中。     

Image-Based Rendering and Synthesis

One of the most important applications in multiview imaging (MVI) is the development of advanced immersive viewing or visualization systems using, for instance, 3DTV. With the introduction of multiview TVs, it is expected that a new age of 3DTV systems will arrive in the near future. Image-based rendering (IBR) refers to a collection of techniques and representations that allow 3-D scenes and objects to be visualized in a realistic way without full 3-D model reconstruction. IBR uses images as the primary substrate. The potential for photorealistic visualization has tremendous appeal, and it has been receiving increasing attention over the years. Applications such as video games, virtual travel, and E-commerce stand to benefit from this technology. This article serves as a tutorial introduction and brief review of this important technology. First the classification, principles, and key research issues of IBR are discussed. Then, an object-based IBR system to illustrate the techniques involved and its potential application in view synthesis and processing are explained. Stereo matching, which is an important technique for depth estimation and view synthesis, is briefly explained and some of the top-ranked methods are highlighted. Finally, the challenging problem of interactive IBR is explained. Possible solutions and some state-of-the-art systems are also reviewed.     

Turning images into 3-D models

In this article developments and performance analysis of image matching for detailed surface reconstruction of heritage objects is discussed. Three dimensional image-based modeling of heritages is a very interesting topic with many possible applications. In this article we propose a multistage image-based modeling approach that requires only a limited amount of human interactivity and is capable of capturing the fine geometric details with similar accuracy as close-range active range sensors. It can also cope with wide baselines using several advancements over standard stereo matching techniques. Our approach is sequential, starting from a sparse basic segmented model created with a small number of interactively measured points. This model, specifically the equation of each surface, is then used as a guide to automatically add the fine details. The following three techniques are used, each where best suited, to retrieve the details: 1) for regularly shaped patches such as planes, cylinders, or quadrics, we apply a fast relative stereo matching technique. 2) For more complex or irregular segments with unknown shape, we use a global multi-image geometrically constrained technique. 3) For segments unsuited for stereo matching, we employ depth from shading (DFS). The goal is not the development of a fully automated procedure for 3D object reconstruction from image data or a sparse stereo approach, but we aim at the digital reconstruction of detailed and accurate surfaces from calibrated and oriented images for practical daily documentation and digital conservation of wide variety of heritage objects.     

Magnetic resonance diffractive imaging

Magnetic resonance (MR) diffractive imaging is proposed as a new approach to MR angiography. The expression of the nuclear MR signal is similar to the equation for the Fresnel diffraction of a three-dimensional (3-D) object in light or sound waves. The proposed technique offers the possibility of fast angiographic imaging and the on-line reconstruction of 3-D volumetric images using the holographic technique. Static imaging experiments using an ultra-low-field MRI system are performed to verify the feasibility of the technique. It is shown that the images focused on an arbitrary plane can be reconstructed from data scanned in two dimensions, even though blurred image data is superimposed on the image. Moreover, the 3-D image can be observed in a coherent optical imaging system. This study demonstrates the possibility of the proposed method as a fast imaging technique for MR angiography.     

Multiscale medial shape-based analysis of image objects

Medial representation of a three-dimensional (3-D) object or an ensemble of 3-D objects involves capturing the object interior as a locus of medial atoms, each atom being two vectors of equal length joined at the tail at the medial point. Medial representation has a variety of beneficial properties, among the most important of which are 1) its inherent geometry, provides an object-intrinsic coordinate system and thus provides correspondence between instances of the object in and near the object(s); 2) it captures the object interior and is, thus, very suitable for deformation; and 3) it provides the basis for an intuitive object-based multiscale sequence leading to efficiency of segmentation algorithms and trainability of statistical characterizations with limited training sets. As a result of these properties, medial representation is particularly suitable for the following image analysis tasks; how each operates will be described and will be illustrated by results: segmentation of objects and object complexes via deformable models; segmentation of tubular trees, e.g., of blood vessels, by following height ridges of measures of fit of medial atoms to target images; object-based image registration via medial loci of such blood vessel trees; statistical characterization of shape differences between control and pathological classes of structures. These analysis tasks are made possible by a new form of medial representation called m-reps, which is described.     

3D Shape Inferencing and Modeling for Video Retrieval

We present a geometry-based indexing approach for the retrieval of video databases. It consists of two modules: 3D object shape inferencing from video data and geometric modeling from the reconstructed shape structure. A motion-based segmentation algorithm employing feature block tracking and principal component split is used for multi-moving-object motion classification and segmentation. After segmentation, feature blocks from each individual object are used to reconstruct its motion and structure through a factorization method. The estimated shape structure and motion parameters are used to generate the implicit polynomial model for the object. The video data is retrieved using the geometric structure of objects and their spatial relationship. We generalize the 2D string to 3D to compactly encode the spatial relationship of objects.     

3-D reconstruction of microcalcification clusters using stereo imaging: algorithm and mammographic unit calibration

The three-dimensional (3-D) shape of microcalcification clusters is an important indicator in early breast cancer detection. In fact, there is a relationship between the cluster topology and the type of lesion (malignant or benign). This paper presents a 3-D reconstruction method for such clusters using two 2-D views acquired during standard mammographic examinations. For this purpose, the mammographic unit was modeled using a camera with virtual optics. This model was used to calibrate the acquisition unit and then to reconstruct the clusters in the 3-D space after microcalcification segmentation and matching. The proposed model is hardware independent since it is suitable for digital mammographic units with different geometries and with various physical acquisition principles. Three-dimensional reconstruction results are presented here to prove the validity of the method. Tests were first performed using a phantom with a well-known geometry. The latter contained X-ray opaque glass balls representing microcalcifications. The positions of these balls were reconstructed with a 16.25-microm mean accuracy. This very high inherent algorithm accuracy is more than enough for a precise 3-D cluster representation. Further validation tests were carried out using a second phantom including a spherical cluster. This phantom was built with materials simulating the behavior of both mammary tissue and microcalcifications toward Xrays. The reconstructed shape was effectively spherical. Finally, reconstructions were carried out for real clusters and their results are also presented.     

Quantitative evaluation of three calibration methods for 3-D freehand ultrasound

In this paper, three different calibration methods for three-dimensional (3-D) freehand ultrasound (US) are evaluated. Calibration is the process of estimating the rigid transformation from US image coordinates to the coordinate system of the tracking sensor mounted onto the probe. Calibration accuracy has an important impact on quantitative studies. Geometrical precision can also be crucial in many interventions and surgery. The proposed evaluation framework relies on a single point phantom and a 3-D US phantom which mimics the US characteristics of human liver. Four quality measures are used: 3-D point localization criterion, distance and volume measurements, and shape based criterion. Results show that during the acquisition procedure, volumetric measurements and shapes of the reconstructed object depend on probe motion used, particularly fan motions for which errors are larger. It is also shown that accurate calibration is essential to obtain reliable quantitative information.     

Exact reduced-complexity maximum likelihood reconstruction of multiple 3-D objects from unlabeled unoriented 2-D projections and electron microscopy of viruses.

In cryo-electron microscopy, the data is comprised of noisy 2-D projection images of the 3-D electron scattering intensity of the object where the orientation of the projections is unknown. Often, the images show randomly selected objects from a mixture of different types of objects. Objects of different type may be unrelated, e.g., different species of virus, or related, e.g., different conformations of the same species of virus. Due to the low SNR and the 2-D nature of the data, it is challenging to determine the type of the object shown in an individual image. A statistical model and maximum likelihood estimator that computes simultaneous 3-D reconstruction and labels using an expectation maximization algorithm exists but requires extensive computation due to the numerical evaluation of 3-D or 5-D integrations of a square matrix of dimension equal to the number of degrees of freedom in the 3-D reconstruction. By exploiting the geometry of rotations in 3-D, the estimation problem can be transformed so that the inner-most numerical integral has a scalar rather than a matrix integrand. This leads to a dramatic reduction in computation, especially as the number of degrees of freedom in the 3-D reconstruction increases. Numerical examples of the 3-D reconstructions are provided based on synthetic and experimental images where the objects are small spherical viruses.