Computation of surface orientation and structure of objects using grid coding

In this correspondence, algorithms are introduced to infer surface orientation and structure of visible object surfaces using grid coding. We adopt the active lighting technique to spatially ``encode' the scene for analysis. The observed objects, which can have surfaces of arbitrary shape, are assumed to rest on a plane (base plane) in a scene which is ``encoded' with light cast through a grid plane. Two orthogonal grid patterns are used, where each pattern is obtained with a set of equally spaced stripes marked on a glass pane. The scene is observed through a camera and the object surface orientation is determined using the projected patterns on the object surface. If the surfaces under consideration obey certain smoothness constraints, a dense orientation map can be obtained through proper interpolation. The surface structure can then be recovered given this dense orientation map. Both planar and curved surfaces can be handled in a uniform manner. The algorithms we propose yield reasonably accurate results and are relatively tolerant to noise, especially when compared to shape-from-shading techniques. In contrast to other grid coding techniques reported which match the grid junctions for depth reconstruction under the stereopsis principle, our techniques use the direction of the projected stripes to infer local surface orientation and do not require any correspondence relationship between either the grid lines or the grid junctions to be specified. The algorithm has the ability to register images and can therefore be embedded in a system which integrates knowledge from multiple views.     

A learning-based visual saliency prediction model for stereoscopic 3D video (LBVS-3D)

Saliency prediction models provide a probabilistic map of relative likelihood of an image or video region to attract the attention of the human visual system. Over the past decade, many computational saliency prediction models have been proposed for 2D images and videos. Considering that the human visual system has evolved in a natural 3D environment, it is only natural to want to design visual attention models for 3D content. Existing monocular saliency models are not able to accurately predict the attentive regions when applied to 3D image/video content, as they do not incorporate depth information. This paper explores stereoscopic video saliency prediction by exploiting both low-level attributes such as brightness, color, texture, orientation, motion, and depth, as well as high-level cues such as face, person, vehicle, animal, text, and horizon. Our model starts with a rough segmentation and quantifies several intuitive observations such as the effects of visual discomfort level, depth abruptness, motion acceleration, elements of surprise, size and compactness of the salient regions, and emphasizing only a few salient objects in a scene. A new fovea-based model of spatial distance between the image regions is adopted for considering local and global feature calculations. To efficiently fuse the conspicuity maps generated by our method to one single saliency map that is highly correlated with the eye-fixation data, a random forest based algorithm is utilized. The performance of the proposed saliency model is evaluated against the results of an eye-tracking experiment, which involved 24 subjects and an in-house database of 61 captured stereoscopic videos. Our stereo video database as well as the eye-tracking data are publicly available along with this paper. Experiment results show that the proposed saliency prediction method achieves competitive performance compared to the state-of-the-art approaches.

     

Conformal geometry and its applications on 3D shape matching, recognition, and stitching

Three-dimensional shape matching is a fundamental issue in computer vision with many applications such as shape registration, 3D object recognition, and classification. However, shape matching with noise, occlusion, and clutter is a challenging problem. In this paper, we analyze a family of quasi-conformal maps including harmonic maps, conformal maps, and least-squares conformal maps with regards to 3D shape matching. As a result, we propose a novel and computationally efficient shape matching framework by using least-squares conformal maps. According to conformal geometry theory, each 3D surface with disk topology can be mapped to a 2D domain through a global optimization and the resulting map is a diffeomorphism, i.e., one-to-one and onto. This allows us to simplify the 3D shape-matching problem to a 2D image-matching problem, by comparing the resulting 2D parametric maps, which are stable, insensitive to resolution changes and robust to occlusion, and noise. Therefore, highly accurate and efficient 3D shape matching algorithms can be achieved by using the above three parametric maps. Finally, the robustness of least-squares conformal maps is evaluated and analyzed comprehensively in 3D shape matching with occlusion, noise, and resolution variation. In order to further demonstrate the performance of our proposed method, we also conduct a series of experiments on two computer vision applications, i.e., 3D face recognition and 3D nonrigid surface alignment and stitching.     

Efficient multiple phase shift patterns for dense 3D acquisition in structured light scanning

Although phase shifts (PS) are frequently used to acquire colored surfaces of static objects, especially when acquisition time is not critical, the periodic nature of relative (wrapped) PS maps makes it necessary to deal with the issue of phase unwrapping. Consequently, multiple phase shifts (MPS) have been widely used as an alternative, but this usually involves a large number of different PS maps to unwrap an absolute (unique) phase. In this paper we propose a new MPS method to unwrap a phase and accurately perform the dense 3D acquisition of neutral and colored objects using only two PS maps. Accuracy is reported including a quantitative and qualitative evaluation of the results.     

Independent motion detection in 3D scenes

This paper presents an algorithmic approach to the problem of detecting independently moving objects in 3D scenes that are viewed under camera motion. There are two fundamental constraints that can be exploited for the problem: 1) two/multiview camera motion constraint (for instance, the epipolar/trilinear constraint) and 2) shape constancy constraint. Previous approaches to the problem either use only partial constraints, or rely on dense correspondences or flow. We employ both the fundamental constraints in an algorithm that does not demand a priori availability of correspondences or flow. Our approach uses the plane-plus-parallax decomposition to enforce the two constraints. It is also demonstrated that for a class of scenes, called sparse 3D scenes in which genuine parallax and independent motions may be confounded, how the plane-plus-parallax decomposition allows progressive introduction, and verification of the fundamental constraints. Results of the algorithm on some difficult sparse 3D scenes are promising.     

Silhouette‐Aware Warping for Image‐Based Rendering

Image‐based rendering (IBR) techniques allow capture and display of 3D environments using photographs. Modern IBR pipelines reconstruct proxy geometry using multi‐view stereo, reproject the photographs onto the proxy and blend them to create novel views. The success of these methods depends on accurate 3D proxies, which are difficult to obtain for complex objects such as trees and cars. Large number of input images do not improve reconstruction proportionally; surface extraction is challenging even from dense range scans for scenes containing such objects. Our approach does not depend on dense accurate geometric reconstruction; instead we compensate for sparse 3D information by variational image warping. In particular, we formulate silhouette‐aware warps that preserve salient depth discontinuities. This improves the rendering of difficult foreground objects, even when deviating from view interpolation. We use a semi‐automatic step to identify depth discontinuities and extract a sparse set of depth constraints used to guide the warp. Our framework is lightweight and results in good quality IBR for previously challenging environments.     

The computation of visible-surface representations

A computational theory of visible-surface representations is developed. The visible-surface reconstruction process that computes these quantitative representations unifies formal solutions to the key problems of: (1) integrating multiscale constraints on surface depth and orientation from multiple-visual sources; (2) interpolating dense, piecewise-smooth surfaces from these constraints; (3) detecting surface depth and orientation discontinuities to apply boundary conditions on interpolation; and (4) structuring large-scale, distributed-surface representations to achieve computational efficiency. Visible-surface reconstruction is an inverse problem. A well-posed variational formulation results from the use of a controlled-continuity surface model. Discontinuity detection amounts to the identification of this generic model's distributed parameters from the data. Finite-element shape primitives yield a local discretization of the variational principle. The result is an efficient algorithm for visible-surface reconstruction     

High Resolution Tracking of Non-Rigid Motion of Densely Sampled 3D Data Using Harmonic Maps

We present a novel automatic method for high resolution, non-rigid dense 3D point tracking. High quality dense point clouds of non-rigid geometry moving at video speeds are acquired using a phase-shifting structured light ranging technique. To use such data for the temporal study of subtle motions such as those seen in facial expressions, an efficient non-rigid 3D motion tracking algorithm is needed to establish inter-frame correspondences. The novelty of this paper is the development of an algorithmic framework for 3D tracking that unifies tracking of intensity and geometric features, using harmonic maps with added feature correspondence constraints. While the previous uses of harmonic maps provided only global alignment, the proposed introduction of interior feature constraints allows to track non-rigid deformations accurately as well. The harmonic map between two topological disks is a diffeomorphism with minimal stretching energy and bounded angle distortion. The map is stable, insensitive to resolution changes and is robust to noise. Due to the strong implicit and explicit smoothness constraints imposed by the algorithm and the high-resolution data, the resulting registration/deformation field is smooth, continuous and gives dense one-to-one inter-frame correspondences. Our method is validated through a series of experiments demonstrating its accuracy and efficiency.     

三维物体空间取向归一化的一种新方法

本文提出三维物体空间取向归一化的新方法，该方法从理论上分析了用主轴方法归一化三维物体空间取向时主轴的唯一性确定问题，本文提出了基于３Ｄ矩的主轴唯一性判别准则解决了由主轴多义性给三维物空间取向归一化带来的困难，我们的方法突破了现有方法只允许物体小角度空间取向变化或者形心必须在物体表面以内的限制条件，基于３Ｄ矩的主轴唯一性判别准则使主轴取向的计算更加稳定，可靠，实验结果表明，对各种取向的物体，利用本文     

First order augmentation to tensor voting for boundary inference and multiscale analysis in 3D

Most computer vision applications require the reliable detection of boundaries. In the presence of outliers, missing data, orientation discontinuities, and occlusion, this problem is particularly challenging. We propose to address it by complementing the tensor voting framework, which was limited to second order properties, with first order representation and voting. First order voting fields and a mechanism to vote for 3D surface and volume boundaries and curve endpoints in 3D are defined. Boundary inference is also useful for a second difficult problem in grouping, namely, automatic scale selection. We propose an algorithm that automatically infers the smallest scale that can preserve the finest details. Our algorithm then proceeds with progressively larger scale to ensure continuity where it has not been achieved. Therefore, the proposed approach does not oversmooth features or delay the handling of boundaries and discontinuities until model misfit occurs. The interaction of smooth features, boundaries, and outliers is accommodated by the unified representation, making possible the perceptual organization of data in curves, surfaces, volumes, and their boundaries simultaneously. We present results on a variety of data sets to show the efficacy of the improved formalism.     

Saliency‐Preserving Slicing Optimization for Effective 3D Printing

We present an adaptive slicing scheme for reducing the manufacturing time for 3D printing systems. Based on a new saliency‐based metric, our method optimizes the thicknesses of slicing layers to save printing time and preserve the visual quality of the printing results. We formulate the problem as a constrained ?₀ optimization and compute the slicing result via a two‐step optimization scheme. To further reduce printing time, we develop a saliency‐based segmentation scheme to partition an object into subparts and then optimize the slicing of each subpart separately. We validate our method with a large set of 3D shapes ranging from CAD models to scanned objects. Results show that our method saves printing time by 30–40% and generates 3D objects that are visually similar to the ones printed with the finest resolution possible.     

A Multi-Channel Salience Based Detail Exaggeration Technique for 3D Relief Surfaces

Visual saliency can always persuade the viewer’s visual attention to fine-scale mesostructure of 3D complex shapes.Owing to the multi-channel salience measure and salience-domain shape modeling technique,a novel visual saliency based shape depiction scheme is presented to exaggerate salient geometric details of the underlying relief surface.Our multi-channel salience measure is calculated by combining three feature maps,i.e.,the 0-order feature map of local height distribution,the 1-order feature map of normal difference,and the 2-order feature map of mean curvature variation.The original relief surface is firstly manipulated by a salience-domain enhancement function,and the detail exaggeration surface can then be obtained by adjusting the surface normals of the original surface as the corresponding final normals of the manipulated surface.The advantage of our detail exaggeration technique is that it can adaptively alter the shading of the original shape to reveal visually salient features whilst keeping the desired appearance unimpaired.The experimental results demonstrate that our non-photorealistic shading scheme can enhance the surface mesostructure effectively and thus improving the shape depiction of the relief surfaces.     

3D shape acquisition and integral compact representation using optical scanning and enhanced shape parameterization

An efficient computational methodology for shape acquisition, processing and representation is developed. It includes 3D computer vision by applying triangulation and stereo-photogrammetry for high-accuracy 3D shape acquisition. Resulting huge 3D point clouds are successively parameterized into mathematical surfaces to provide for compact data-set representation, yet capturing local details sufficiently. B-spline surfaces are employed as parametric entities in fitting to point clouds resulting from optical 3D scanning. Beyond the linear best-fitting algorithm with control points as fitting variables, an enhanced non-linear procedure is developed. The set of best fitting variables in minimizing the approximation error norm between the parametric surface and the 3D cloud includes the control points coordinates. However, they are augmented by the set of position parameter values which identify the respectively closest matching points on the surface for the points in the cloud. The developed algorithm is demonstrated to be efficient on demanding test cases which encompass sharp edges and slope discontinuities originating from physical damage of the 3D objects or shape complexity.     

Driving 3D morphable models using shading cues

In this paper we show how surface orientation information inferred using shape-from-shading can be used to aid the process of fitting a 3D morphable model to an image of a face. We consider the problem of model dominance and show how shading constraints can be used to refine morphable model shape estimates, offering the possibility of exceeding the maximum possible accuracy of the model. We use this observation to motivate an optimisation scheme based on surface normal error. This ensures the fullest possible use of the information conveyed by the shading in an image. Moreover, our framework allows estimation of per-vertex albedo and bump maps which are not constrained to lie within the span of the model. This means the recovered model is capable of describing shape and reflectance phenomena not present in the training set. We show reconstruction and synthesis results and demonstrate that the shape and albedo estimates can be used for illumination insensitive recognition using only a single gallery image.     

Perception of 3-D surfaces from 2-D contours

Inference of 3-D shape from 2-D contours in a single image is an important problem in machine vision. The authors survey classes of techniques proposed in the past and provide a critical analysis. They show that two kinds of symmetries in figures, which are known as parallel and skew symmetries, give significant information about surface shape for a variety of objects. They derive the constraints imposed by these symmetries and show how to use them to infer 3-D shape. They also discuss the zero Gaussian curvature (ZGC) surfaces in depth and show results on the recovery of surface orientation for various ZGC surfaces     

Capture and fusion of 3d surface texture

Image fusion is a process that multiple images of a scene are combined to form a single image. The aim of image fusion is to preserve the full content and retain important features of each original image. In this paper, we propose a novel approach based on wavelet transform to capture and fusion of real-world rough surface textures, which are commonly used in multimedia applications and referred to as3D surface texture. These textures are different from 2D textures as their appearances can vary dramatically with different illumination conditions due to complex surface geometry and reflectance properties. In our approach, we first extract gradient/height and albedo maps from sample 3D surface texture images as their representation. Then we measure saliency of wavelet coefficients of these 3D surface texture representations. The saliency values reflect the meaningful content of the wavelet coefficients and are consistent with human visual perception. Finally we fuse the gradient/height and albedo maps based on the measured saliency values. This novel scheme aims to preserve the original texture patterns together with geometry and reflectance characteristics from input images. Experimental results show that the proposed approach can not only capture and fuse 3D surface texture under arbitrary illumination directions, but also has the ability to retain the surface geometry properties and preserve perceptual features in the original images.     

3D SSD tracking with estimated 3D planes

We present a tracking method where full camera position and orientation is tracked from intensity differences in a video sequence. The camera pose is calculated based on 3D planes, and hence does not depend on point correspondences. The plane based formulation also allows additional constraints to be naturally added, e.g., perpendicularity between walls, floor and ceiling surfaces, co-planarity of wall surfaces etc. A particular feature of our method is that the full 3D pose change is directly computed from temporal image differences without making a commitment to a particular intermediate (e.g., 2D feature) representation. We experimentally compared our method with regular 2D SSD tracking and found it more robust and stable. This is due to 3D consistency being enforced even in the low level registration of image regions. This yields better results than first computing (and hence committing to) 2D image features and then from these compute 3D pose.     

High Reliefs from 3D Scenes

We present a method for synthesizing high reliefs, a sculpting technique that attaches 3D objects onto a 2D surface within a limited depth range. The main challenges are the preservation of distinct scene parts by preserving depth discontinuities, the fine details of the shape, and the overall continuity of the scene. Bas relief depth compression methods such as gradient compression and depth range compression are not applicable for high relief production. Instead, our method is based on differential coordinates to bring scene elements to the relief plane while preserving depth discontinuities and surface details of the scene. We select a user‐defined number of attenuation points within the scene, attenuate these points towards the relief plane and recompute the positions of all scene elements by preserving the differential coordinates. Finally, if the desired depth range is not achieved we apply a range compression. High relief synthesis is semi‐automatic and can be controlled by user‐defined parameters to adjust the depth range, as well as the placement of the scene elements with respect to the relief plane.