The convergence of graphics and vision

Computer graphics and computer vision are inverse problems. Traditional computer graphics starts with input geometric models and produces image sequences. Traditional computer vision starts with input image sequences and produces geometric models. Lately, there has been a meeting in the middle, and the center, the prize, is to create stunning images in real time. Vision researchers now work from images backward, just as far backward as necessary to create models that capture a scene without going to full geometric models. Graphics researchers now work with hybrid geometry and image models. Approaching similar problems from opposite directions, graphics and vision researchers are reaching a fertile middle ground. The goal is to find the best possible tools for the imagination. This overview describes cutting edge work, some of which will debut at Siggraph 98     

Web-Based Remote Renderingwith IBRAC (Image-Based Rendering Acceleration and Compression)

Recent advances in Internet and computer graphics stimulate intensive use and development of 3D graphics on the World Wide Web. To increase efficiency of systems using 3D graphics on the web, the presented method utilizes previously rendered and transmitted images to accelerate the rendering and compression of new synthetic scene images. The algorithm employs ray casting and epipolar constraints to exploit spatial and temporal coherence between the current and previously rendered images. The reprojection of color and visibility data accelerates the computation of new images. The rendering method intrinsically computes a residual image, based on a user specified error tolerance that balances image quality against computation time and bandwidth. Encoding and decoding uses the same algorithm, so the transmitted residual image consists only of significant data without addresses or offsets. We measure rendering speed-ups of four to seven without visible degradation. Compression ratios per frame are a factor of two to ten better than MPEG2 in our test cases. There is no transmission of 3D scene data to delay the first image. The efficiency of the server and client generally increases with scene complexity or data size since the rendering time is predominantly a function of image size. This approach is attractive for remote rendering applications such as web-based scientific visualization where a client system may be a relatively low-performance machine and limited network bandwidth makes transmission of large 3D data impractical.     

N-dimensional tensor voting and application to epipolar geometryestimation

We address the problem of epipolar geometry estimation by formulating it as one of hyperplane inference from a sparse and noisy point set in an 8D space. Given a set of noisy point correspondences in two images of a static scene without correspondences, even in the presence of moving objects, our method extracts good matches and rejects outliers. The methodology is novel and unconventional, since, unlike most other methods optimizing certain scalar, objective functions, our approach does not involve initialization or any iterative search in the parameter space. Therefore, it is free of the problem of local optima or poor convergence. Further, since no search is involved, it is unnecessary to impose simplifying assumption to the scene being analyzed for reducing the search complexity. Subject to the general epipolar constraint only, we detect wrong matches by a computation scheme, 8D tensor voting, which is an instance of the more general N-dimensional tensor voting framework. In essence, the input set of matches is first transformed into a sparse 8D point set. Dense, 8D tensor kernels are then used to vote for the most salient hyperplane that captures all inliers inherent in the input. With this filtered set of matches, the normalized eight-point algorithm can be used to estimate the fundamental matrix accurately. By making use of efficient data structure and locality, our method is both time and space efficient despite the higher dimensionality. We demonstrate the general usefulness of our method using example image pairs for aerial image analysis, with widely different views, and from nonstatic 3D scenes. Each example contains a considerable number of wrong matches     

Face recognition based on fitting a 3D morphable model

This paper presents a method for face recognition across variations in pose, ranging from frontal to profile views, and across a wide range of illuminations, including cast shadows and specular reflections. To account for these variations, the algorithm simulates the process of image formation in 3D space, using computer graphics, and it estimates 3D shape and texture of faces from single images. The estimate is achieved by fitting a statistical, morphable model of 3D faces to images. The model is learned from a set of textured 3D scans of heads. We describe the construction of the morphable model, an algorithm to fit the model to images, and a framework for face identification. In this framework, faces are represented by model parameters for 3D shape and texture. We present results obtained with 4,488 images from the publicly available CMU-PIE database and 1,940 images from the FERET database.     

Capture and fusion of 3d surface texture

Image fusion is a process that multiple images of a scene are combined to form a single image. The aim of image fusion is to preserve the full content and retain important features of each original image. In this paper, we propose a novel approach based on wavelet transform to capture and fusion of real-world rough surface textures, which are commonly used in multimedia applications and referred to as3D surface texture. These textures are different from 2D textures as their appearances can vary dramatically with different illumination conditions due to complex surface geometry and reflectance properties. In our approach, we first extract gradient/height and albedo maps from sample 3D surface texture images as their representation. Then we measure saliency of wavelet coefficients of these 3D surface texture representations. The saliency values reflect the meaningful content of the wavelet coefficients and are consistent with human visual perception. Finally we fuse the gradient/height and albedo maps based on the measured saliency values. This novel scheme aims to preserve the original texture patterns together with geometry and reflectance characteristics from input images. Experimental results show that the proposed approach can not only capture and fuse 3D surface texture under arbitrary illumination directions, but also has the ability to retain the surface geometry properties and preserve perceptual features in the original images.     

TransCAIP: A Live 3D TV System Using a Camera Array and an Integral Photography Display with Interactive Control of Viewing Parameters

The system described in this paper provides a real-time 3D visual experience by using an array of 64 video cameras and an integral photography display with 60 viewing directions. The live 3D scene in front of the camera array is reproduced by the full-color, full-parallax autostereoscopic display with interactive control of viewing parameters. The main technical challenge is fast and flexible conversion of the data from the 64 multicamera images to the integral photography format. Based on image-based rendering techniques, our conversion method first renders 60 novel images corresponding to the viewing directions of the display, and then arranges the rendered pixels to produce an integral photography image. For real-time processing on a single PC, all the conversion processes are implemented on a GPU with GPGPU techniques. The conversion method also allows a user to interactively control viewing parameters of the displayed image for reproducing the dynamic 3D scene with desirable parameters. This control is performed as a software process, without reconfiguring the hardware system, by changing the rendering parameters such as the convergence point of the rendering cameras and the interval between the viewpoints of the rendering cameras.     

Multi-Perspective Modelling, Rendering and Imaging

A perspective image represents the spatial relationships of objects in a scene as they appear from a single viewpoint. In contrast, a multi‐perspective image combines what is seen from several viewpoints into a single image. Despite their incongruity of view, effective multi‐perspective images are able to preserve spatial coherence and can depict, within a single context, details of a scene that are simultaneously inaccessible from a single view, yet easily interpretable by a viewer. In computer vision, multi‐perspective images have been used for analysing structure revealed via motion and generating panoramic images with a wide field‐of‐view using mirrors. In this STAR, we provide a practical guide on topics in multi‐perspective modelling and rendering methods and multi‐perspective imaging systems. We start with a brief review of multi‐perspective image techniques frequently employed by artists such as the visual paradoxes of Escher, the Cubism of Picasso and Braque and multi‐perspective panoramas in cel animations. We then characterize existing multi‐perspective camera models, with an emphasis on their underlying geometry and image properties. We demonstrate how to use these camera models for creating specific multi‐perspective rendering effects. Furthermore, we show that many of these cameras satisfy the multi‐perspective stereo constraints and we demonstrate several multi‐perspective imaging systems for extracting 3D geometry for computer vision. The participants learn about topics in multi‐perspective modelling and rendering for generating compelling pictures for computer graphics and in multi‐perspective imaging for extracting 3D geometry for computer vision. We hope to provide enough fundamentals to satisfy the technical specialist without intimidating curious digital artists interested in multi‐perspective images. The intended audience includes digital artists, photographers and computer graphics and computer vision researchers using or building multi‐perspective cameras. They will learn about multi‐perspective modelling and rendering, along with many real world multi‐perspective imaging systems.     

Quality time-of-flight range imaging for feature-based registration using bacterial foraging

Image registration is a widely tackled research topic in the computer vision and the computer graphics fields. This problem aims to find an optimal transformation or correspondence between images acquired under different conditions. Recently, a new 3D image acquisition device based on the time-of-flight technology has appeared which obtains range images from real-time 3D video sequences. In this contribution, we aim to study the feasibility of using this new class of cameras to face the 3D model reconstruction procedure. Our proposal is two-fold. First, we introduce a novel image preprocessing pipeline in order to improve the quality of time-of-flight range images and a subsequent feature extraction method considering both 2D and 3D images. As second major objective, we propose an adaptation of the evolutionary bacterial foraging optimization algorithm, which has recently emerged as a very powerful technique for real parameter optimization and gained a high interest for distributed optimization and control, to tackle the range image registration problem. Finally, we analyse the performance of our proposal against other state-of-the-art evolutionary image registration methods.     

Illumination normalization of facial images by reversing the process of image formation

Variations in illumination degrade the performance of appearance based face recognition. We present a novel algorithm for the normalization of color facial images using a single image and its co-registered 3D pointcloud (3D image). The algorithm borrows the physically based Phong’s lighting model from computer graphics which is used for rendering computer images and employs it in a reverse mode for the calculation of face albedo from real facial images. Our algorithm estimates the number of the dominant light sources and their directions from the specularities in the facial image and the corresponding 3D points. The intensities of the light sources and the parameters of the Phong’s model are estimated by fitting the Phong’s model onto the facial skin data. Unlike existing approaches, our algorithm takes into account both Lambertian and specular reflections as well as attached and cast shadows. Moreover, our algorithm is invariant to facial pose and expression and can effectively handle the case of multiple extended light sources. The algorithm was tested on the challenging FRGC v2.0 data and satisfactory results were achieved. The mean fitting error was 6.3% of the maximum color value. Performing face recognition using the normalized images increased both identification and verification rates.     

基于图像的室内虚拟环境的研究

基于图像的建模和绘制技术,提出了一个构造室内虚拟环境的完整方案,用户只需要输入少数照片,即可重建室内场景的全景图像,方案主要包括以下几点：首先由用户交互确定图像中的匹配象素,通过运动分析算法恢复整个场景的几何结构,然后,将原始图像变换至平面的参数坐标系,抽取纹理图像,并在参数空间对纹理图像进行拼接;最后生成场景的全景图像,算法对拍摄条件和设备没有苛刻要求,运算量较小,有较强的稳定性。     

基于二维图像的三维人脸建模技术

建立三维人脸模型和表情动画是计算机图形学领域的一个研究热点。文章提出了一种基于二维图像的三维人脸建模方法,首先在给定的人脸的正侧面照片上提取事先定义好的反映人脸特征的特征点信息,与一个一般人脸模型上对应点的信息进行比较和修改,得到反映给定人脸特征的特定人脸模型。最后使用纹理映射技术给特定人脸模型添加纹理信息,形成真实感的虚拟三维人脸模型。     

Comprehensive Halftoning of 3D Scenes

The display of images on binary output hardware requires a halftoning step. Conventional halftoning algorithms approximate image values independently from the image content and often introduce artificial texture that obscures fine details. The objective of this research is to adapt a halftoning technique to 3D scene information and thus to enhance the display of computer generated 3D scenes. Our approach is based on the control of halftoning texture by the combination of ordered dithering and error diffusion techniques. We extend our previous work and enable a user to specify the shape, scale, direction, and contrast of the halftoning texture using an external buffer. We control texture shape by constructing a dither matrix from an arbitrary image or a procedural texture. Texture direction and scale are adapted to the external information by the mapping function. Texture contrast and the accuracy of tone reproduction are varied across the image using the error diffusion process. We halftone images of 3D scenes by using the geometry, position, and illumination information to control the halftoning texture. Thus, the texture provides visual cues and can be used to enhance the viewer's comprehension of the display.     

基于IGES的三维图形浏览器

介绍了一个基于IGES的三维图形浏览器Net3DBrowser的设计及实现。该系统将IGES作为数据交换标准,不同CAD系统产生的三维图形文件在转化为IGES格式后,均可通过该系统进行浏览;它具有良好的用户接口,用户可方便地操作场景中的三维实体。     

基于GPU带有复杂边界的三维实时流体模拟

在GPU(graphics processing unit)上求解了复杂场景中的三维流动问题,充分利用了GPU并行能力以加速计算.与前人的方法不同,该方法对于边界条件的处理更为通用.首先,通过在图像空间生成实心的剖切截面构成整个障碍物信息图,算法使得流体计算与整个几何场景的复杂度无关,通过对各体素进行分类并结合边界条件,根据障碍物形成修正因子来修改对应的值;另外,采用更为紧凑的数据格式,以充分利用硬件的并行性.通过将所有标量的运算压缩到纹元的4个颜色通道并结合平铺三维纹理,减少了三维流场计算所需要的绘制次数.实验结果显示出算法的有效性和高效率.该算法可以实时计算并显示一个采用中等规模离散的复杂场景.     

Automatic Reconstruction of Buildings from Stereoscopic Image Sequences

A vision–based 3-D scene analysis system is described that is capable to model complex real–world scenes like streets and buildings automatically from stereoscopic image pairs. Input to the system is a sequence of stereoscopic images taken with two standard CCD Cameras and TV lenses. The relative orientation of both cameras to each other is known by calibration. The camerapair is then moved throughout the scene and a long sequence of closely spaced views is recorded. Each of the stereoscopic image pairs is rectified and a dense map of 3-D suface points is obtained by area correlation, object segmentation, interpolation, and triangulation. 3-D camera motion relative to the scene coordinate system is tracked directly from the image sequence which allows to fuse 3-D surface measurements from different viewpoints into a consistent 3-D model scene. The surface geometry of each scene object is approximated by a triangular surface mesh which stores the suface texture in a texture map. From the textured 3-D models, realistic looking image sequences from arbitrary view points can be synthesized using computer graphics.     

3D face computational photography using PCA spaces

In this paper, we present a 3D face photography system based on a facial expression training dataset, composed of both facial range images (3D geometry) and facial texture (2D photography). The proposed system allows one to obtain a 3D geometry representation of a given face provided as a 2D photography, which undergoes a series of transformations through the texture and geometry spaces estimated. In the training phase of the system, the facial landmarks are obtained by an active shape model (ASM) extracted from the 2D gray-level photography. Principal components analysis (PCA) is then used to represent the face dataset, thus defining an orthonormal basis of texture and another of geometry. In the reconstruction phase, an input is given by a face image to which the ASM is matched. The extracted facial landmarks and the face image are fed to the PCA basis transform, and a 3D version of the 2D input image is built. Experimental tests using a new dataset of 70 facial expressions belonging to ten subjects as training set show rapid reconstructed 3D faces which maintain spatial coherence similar to the human perception, thus corroborating the efficiency and the applicability of the proposed system.     

Special relativistic visualization by local ray tracing

Special relativistic visualization offers the possibility of experiencing the optical effects of traveling near the speed of light, including apparent geometric distortions as well as Doppler and searchlight effects. Early high-quality computer graphics images of relativistic scenes were created using offline, computationally expensive CPU-side 4D ray tracing. Alternate approaches such as image-based rendering and polygon-distortion methods are able to achieve interactivity, but exhibit inferior visual quality due to sampling artifacts. In this paper, we introduce a hybrid rendering technique based on polygon distortion and local ray tracing that facilitates interactive high-quality visualization of multiple objects moving at relativistic speeds in arbitrary directions. The method starts by calculating tight image-space footprints for the apparent triangles of the 3D scene objects. The final image is generated using a single image-space ray tracing step incorporating Doppler and searchlight effects. Our implementation uses GPU shader programming and hardware texture filtering to achieve high rendering speed.     

Visualizing features and tracking their evolution

We describe basic algorithms to extract coherent amorphous regions (features or objects) from 2 and 3D scalar and vector fields and then track them in a series of consecutive time steps. We use a combination of techniques from computer vision, image processing, computer graphics, and computational geometry and apply them to data sets from computational fluid dynamics. We demonstrate how these techniques can reduce visual clutter and provide the first step to quantifying observable phenomena. These results can be generalized to other disciplines with continuous time-dependent scalar (and vector) fields     

基于3D WARP的混合绘制算法研究

几何绘制（GBR）是传统计算机图形学的研究内容,随着计算机图形学的发展,图象绘制（IBR）成为研究热点,两者各有其优缺点,其中,GBR具有绘制自由度大,速度慢等特点,需占用大量系统资源,而IBR则正好相反,因此,寻找一种能够结合GBR与IBR优点,而又能克服其缺点的方法是必要的,为了使几何绘制和图象绘制能够实现有机地结合,提出了一 种全新的“基于几何与图象的混合绘制（Geometry and Image-Based Hybrid Rendering,GIBHR)”方法,并分析了GIBHR的研究内容,同时给出了GIBHR的绘制流程,在此基础上,将IBR领域的3DWARP 法在几何绘制领域进行了进一步拓展与推论,证明了其可行性和有效性,提出了一种基于3DWARP的混合绘制算法,最后对算法进行了实验,结果表明,该算法在提高几何绘制速度方面具有明显优势,且具有空间反走样能力。     

Intrinsic Decompositions for Image Editing

Intrinsic images are a mid‐level representation of an image that decompose the image into reflectance and illumination layers. The reflectance layer captures the color/texture of surfaces in the scene, while the illumination layer captures shading effects caused by interactions between scene illumination and surface geometry. Intrinsic images have a long history in computer vision and recently in computer graphics, and have been shown to be a useful representation for tasks ranging from scene understanding and reconstruction to image editing. In this report, we review and evaluate past work on this problem. Specifically, we discuss each work in terms of the priors they impose on the intrinsic image problem. We introduce a new synthetic ground‐truth dataset that we use to evaluate the validity of these priors and the performance of the methods. Finally, we evaluate the performance of the different methods in the context of image‐editing applications.