Stereo reconstruction from multiperspective panoramas

A new approach to computing a panoramic (360 degrees) depth map is presented in this paper. Our approach uses a large collection of images taken by a camera whose motion has been constrained to planar concentric circles. We resample regular perspective images to produce a set of multiperspective panoramas and then compute depth maps directly from these resampled panoramas. Our panoramas sample uniformly in three dimensions: rotation angle, inverse radial distance, and vertical elevation. The use of multiperspective panoramas eliminates the limited overlap present in the original input images and, thus, problems as in conventional multibaseline stereo can be avoided. Our approach differs from stereo matching of single-perspective panoramic images taken from different locations, where the epipolar constraints are sine curves. For our multiperspective panoramas, the epipolar geometry, to the first order approximation, consists of horizontal lines. Therefore, any traditional stereo algorithm can be applied to multiperspective panoramas with little modification. In this paper, we describe two reconstruction algorithms. The first is a cylinder sweep algorithm that uses a small number of resampled multiperspective panoramas to obtain dense 3D reconstruction. The second algorithm, in contrast, uses a large number of multiperspective panoramas and takes advantage of the approximate horizontal epipolar geometry inherent in multiperspective panoramas. It comprises a novel and efficient 1D multibaseline matching technique, followed by tensor voting to extract the depth surface. Experiments show that our algorithms are capable of producing comparable high quality depth maps which can be used for applications such as view interpolation.     

Multi-Perspective Modelling, Rendering and Imaging

A perspective image represents the spatial relationships of objects in a scene as they appear from a single viewpoint. In contrast, a multi‐perspective image combines what is seen from several viewpoints into a single image. Despite their incongruity of view, effective multi‐perspective images are able to preserve spatial coherence and can depict, within a single context, details of a scene that are simultaneously inaccessible from a single view, yet easily interpretable by a viewer. In computer vision, multi‐perspective images have been used for analysing structure revealed via motion and generating panoramic images with a wide field‐of‐view using mirrors. In this STAR, we provide a practical guide on topics in multi‐perspective modelling and rendering methods and multi‐perspective imaging systems. We start with a brief review of multi‐perspective image techniques frequently employed by artists such as the visual paradoxes of Escher, the Cubism of Picasso and Braque and multi‐perspective panoramas in cel animations. We then characterize existing multi‐perspective camera models, with an emphasis on their underlying geometry and image properties. We demonstrate how to use these camera models for creating specific multi‐perspective rendering effects. Furthermore, we show that many of these cameras satisfy the multi‐perspective stereo constraints and we demonstrate several multi‐perspective imaging systems for extracting 3D geometry for computer vision. The participants learn about topics in multi‐perspective modelling and rendering for generating compelling pictures for computer graphics and in multi‐perspective imaging for extracting 3D geometry for computer vision. We hope to provide enough fundamentals to satisfy the technical specialist without intimidating curious digital artists interested in multi‐perspective images. The intended audience includes digital artists, photographers and computer graphics and computer vision researchers using or building multi‐perspective cameras. They will learn about multi‐perspective modelling and rendering, along with many real world multi‐perspective imaging systems.     

Mental imagery for a conversational robot

To build robots that engage in fluid face-to-face spoken conversations with people, robots must have ways to connect what they say to what they see. A critical aspect of how language connects to vision is that language encodes points of view. The meaning of my left and your left differs due to an implied shift of visual perspective. The connection of language to vision also relies on object permanence. We can talk about things that are not in view. For a robot to participate in situated spoken dialog, it must have the capacity to imagine shifts of perspective, and it must maintain object permanence. We present a set of representations and procedures that enable a robotic manipulator to maintain a "mental model" of its physical environment by coupling active vision to physical simulation. Within this model, "imagined" views can be generated from arbitrary perspectives, providing the basis for situated language comprehension and production. An initial application of mental imagery for spatial language understanding for an interactive robot is described.     

基于全方位视觉传感器倒车辅助装置的设计

倒车所遇到的窘境越来越多地受到人们的关注,倒车的成功与否很大程度上取决于驾驶员对车辆周围情况的了解;设计了一种基于全方位视觉传感器的倒车辅助装置,通过全方位摄像装置采集车辆周围360°的图像信息,运用全方位透视展开算法将全方位图像在嵌入式系统中展开成前后左右4个视角的透视图像,并显示在同一个液晶显示屏上;该装置减轻了驾驶者的体力、脑力劳动强度,降低了倒车难度,避免驾驶员因方向感不强、判断和操作失误而引起的事故,从而达到了安全辅助倒车的目的。     

Looking Around the Corner using Ultrafast Transient Imaging

We propose a novel framework called transient imaging for image formation and scene understanding through impulse illumination and time images. Using time-of-flight cameras and multi-path analysis of global light transport, we pioneer new algorithms and systems for scene understanding through time images. We demonstrate that our proposed transient imaging framework allows us to accomplish tasks that are well beyond the reach of existing imaging technology. For example, one can infer the geometry of not only the visible but also the hidden parts of a scene, enabling us to look around corners. Traditional cameras estimate intensity per pixel I(x,y). Our transient imaging camera captures a 3D time-image I(x,y,t) for each pixel and uses an ultra-short pulse laser for illumination. Emerging technologies are supporting cameras with a temporal-profile per pixel at picosecond resolution, allowing us to capture an ultra-high speed time-image. This time-image contains the time profile of irradiance incident at a sensor pixel. We experimentally corroborated our theory with free space hardware experiments using a femtosecond laser and a picosecond accurate sensing device. The ability to infer the structure of hidden scene elements, unobservable by both the camera and illumination source, will create a range of new computer vision opportunities.     

Increasing the Spatial Resolution of BTF Measurement with Scheimpflug Imaging

We present an improved way of acquiring spatially varying surface reflectance represented by a bidirectional texture function (BTF). Planar BTF samples are measured as images at several inclination angles which puts constraints on the minimum depth of field of cameras used in the measurement instrument. For standard perspective imaging, we show that the size of a sample measured and the achievable spatial resolution are strongly interdependent and limited by diffraction at the lens' aperture. We provide a formula for this relationship. We overcome the issue of the required depth of field by using Scheimpflug imaging further enhanced by an anamorphic attachment. The proposed optics doubles the spatial resolution of images compared to standard perspective imaging optics. We built an instrument prototype with the proposed optics that is portable and allows for measurement on site. We show rendered images using the visual appearance measured by the instrument prototype.     

Implementation and analysis of an image-based global illuminationframework for animated environments

We describe a new framework for efficiently computing and storing global illumination effects for complex, animated environments. The new framework allows the rapid generation of sequences representing any arbitrary path in a “view space” within an environment in which both the viewer and objects move. The global illumination is stored as time sequences of range-images at base locations that span the view space. We present algorithms for determining locations for these base images, and the time steps required to adequately capture the effects of object motion. We also present algorithms for computing the global illumination in the base images that exploit spatial and temporal coherence by considering direct and indirect illumination separately. We discuss an initial implementation using the new framework. Results and analysis of our implementation demonstrate the effectiveness of the individual phases of the approach; we conclude with an application of the complete framework to a complex environment that includes object motion     

Double-stream Convolutional Neural Networks for Machine Vision Inspection of Natural Products

There are known applications of convolutional neural networks to vision inspection of natural products. For many products it is sufficient to acquire and process a single image, but some might require imaging from two sides. Human experts performing quality inspection of malting barley typically only observe one side of each grain, but in doubtful cases look at both sides, intrinsically combining the information. In this paper, we make two contributions. We present a method for determining whether imaging objects from two sides yields performance benefits over single-sided imaging. Then we introduce a double-stream convolutional network for reasoning from two images simultaneously and analyze several methods of combining information from two streams. We find that when orientation of the object is unpredictable and the streams are not specialized to process a particular view, a fully shared architecture combining information on the prediction level yields best performance (98.7% accuracy on our dataset).     

Neuromatrix: The World is Embodied in our Brain

Our brain is an interpreter, probably the most powerful ever conceived by Nature. However, just like every interpreter, it is limited and sometimes it can get things wrong. Most of us take it for granted that what we touch, hear, see, taste and smell is “reality”. But this reality is an illusion, a construct, which only exists in our brain, that is, a mental representation. Indeed, what if we could perceive infrared radiations like Cobras or Predator? What if we could see in very low light like cats? What if we could smell people and foods hidden from view 500 yards away? Surely “reality” would feel much different to us if our sensors and brain were able to extract and process such information from the external world. Even more puzzling is the fact that our senses can betray us completely. This is what happens when we see things that are not there or, on the contrary, when we do not see things that are before our eyes. During this lecture, I will attempt to show how the world in embodied in our brain by looking at examples of interactive visual illusions, by showing the power of attention in defining what we see in the world around us, and by introducing patients with brain lesions or mental illness who perceive the world differently.     

全方位图像展开成全景和透视图的实现方法

全方位视觉系统生成的全方位图像能获得水平方向3600环境信息,非常适合于某些有大视场需求的应用。该文介绍了全方位图像的特点、生成原理及其展开成全景图及透视图的具体实现方法。同时也介绍了以DirectShow为工具,全方位视频的展开实现方法。最后通过实验证明了全景图及透视图生成方法的有效性。     

TORNADO: omnistereo video imaging with rotating optics

One of the key techniques for vision-based communication is omnidirectional stereo (omnistereo) imaging, in which stereoscopic images for an arbitrary horizontal direction are captured and presented according to the viewing direction of the observer. Although omnistereo models have been surveyed in several studies, few omnistereo sensors have actually been implemented. In this paper, a practical method for capturing omnistereo video sequences using rotating optics is proposed and evaluated. The rotating optics system consists of prism sheets, circular or linear polarizing films, and a hyperboloidal mirror. This system has two different modes of operation with regard to the separation of images for the left and right eyes. In the high-speed shutter mode, images are separated using postimage processing, while, in the low-speed shutter mode, the image separation is completed by optics. By capturing actual images, we confirmed the effectiveness of the methods.     

Programmable Imaging: Towards a Flexible Camera

In this paper, we introduce the notion of a programmable imaging system. Such an imaging system provides a human user or a vision system significant control over the radiometric and geometric characteristics of the system. This flexibility is achieved using a programmable array of micro-mirrors. The orientations of the mirrors of the array can be controlled with high precision over space and time. This enables the system to select and modulate rays from the scene’s light field based on the needs of the application at hand. We have implemented a programmable imaging system that uses a digital micro-mirror device (DMD), which is used in digital light processing. Although the mirrors of this device can only be positioned in one of two states, we show that our system can be used to implement a wide variety of imaging functions, including, high dynamic range imaging, feature detection, and object recognition. We also describe how a micro-mirror array that allows full control over the orientations of its mirrors can be used to instantly change the field of view and resolution characteristics of the imaging system. We conclude with a discussion on the implications of programmable imaging for computer vision.     

Performance Capture of High‐Speed Motion Using Staggered Multi‐View Recording

We present a markerless performance capture system that can acquire the motion and the texture of human actors performing fast movements using only commodity hardware. To this end we introduce two novel concepts: First, a staggered surround multi‐view recording setup that enables us to perform model‐based motion capture on motion‐blurred images, and second, a model‐based deblurring algorithm which is able to handle disocclusion, self‐occlusion and complex object motions. We show that the model‐based approach is not only a powerful strategy for tracking but also for deblurring highly complex blur patterns.     

A Theory of Single-Viewpoint Catadioptric Image Formation

Conventional video cameras have limited fields of view which make them restrictive for certain applications in computational vision. A catadioptric sensor uses a combination of lenses and mirrors placed in a carefully arranged configuration to capture a much wider field of view. One important design goal for catadioptric sensors is choosing the shapes of the mirrors in a way that ensures that the complete catadioptric system has a single effective viewpoint. The reason a single viewpoint is so desirable is that it is a requirement for the generation of pure perspective images from the sensed images. In this paper, we derive the complete class of single-lens single-mirror catadioptric sensors that have a single viewpoint. We describe all of the solutions in detail, including the degenerate ones, with reference to many of the catadioptric systems that have been proposed in the literature. In addition, we derive a simple expression for the spatial resolution of a catadioptric sensor in terms of the resolution of the cameras used to construct it. Moreover, we include detailed analysis of the defocus blur caused by the use of a curved mirror in a catadioptric sensor.     

双视三维重建的高精度运动参数估计方法

从两幅透视图像恢复被摄目标的三维结构是计算机视觉最基本的任务之一,其中,运动估计算法的性能决定了最终的三维重建精度。首先讨论了双视成像的基本数学模型,并介绍了几种现有运动参数估计方法的基本原理和不足。随后,基于投影误差最小判决函数,提出了用于双像运动估计的改进非线性迭代优化方法。数值仿真结果表明,在大平移小旋转角及小平移大旋转角2种运动条件下,采用文中提出的方法,运动估计精度均有所提高。此外,根据运动参数的估计值对真实目标进行三维重建实验,结果表明尺度重建误差小于2%且角度误差在3°以内。     

A computational model of view degeneracy

We quantify the observation by Kender and Freudenstein (1987) that degenerate views occupy a significant fraction of the viewing sphere surrounding an object. For a perspective camera geometry, we introduce a computational model that can be used to estimate the probability that a view degeneracy will occur in a random view of a polyhedral object. For a typical recognition system parameterization, view degeneracies typically occur with probabilities of 20 percent and, depending on the parameterization, as high as 50 percent. We discuss the impact of view degeneracy on the problem of object recognition and, for a particular recognition framework, relate the cost of object disambiguation to the probability of view degeneracy. To reduce this cost, we incorporate our model of view degeneracy in an active focal length control paradigm that balances the probability of view degeneracy with the camera field of view. In order to validate both our view degeneracy model as well as our active focal length control model, a set of experiments are reported using a real recognition system operating on real images     

Representation of Pseudo Inter-reflection and Transparency by Considering Characteristics of Human Vision

We have succeeded in developing a quick and fully automated system that can generate photo‐realistic 3D CG data based on a real object. A major factor in this success comes from our findings through psychophysical experiments that human observers do not have an accurate idea of what should be actually reflected as inter‐reflections on the surface of an object. Taking advantage of this characteristic of human vision, we propose a new inter‐reflection representation technique in which inter‐reflections are simulated by allowing the same quantity of reflection components as there are in the background to pass through the object. Since inter‐reflection and transparency are calculated by the same algorithm, our system can capture 3D CG data from various real objects having a strong inter‐reflection, such as plastic and porcelain items or translucent glass and acrylic resin objects. The synthetic images from the 3D CG data generated with this pseudo inter‐reflection and transparency look very natural. In addition, the 3D CG data and synthetic images are produced quickly at a lower cost.     

Optical simulation for cross‐talk evaluation and improvement of autostereoscopic 3‐D displays with a projector array

Abstract— Multi‐view spatial‐multiplexed autostereoscopic 3‐D displays normally use a 2‐D image source and divide the pixels to generate perspective images. Due to the reduction in the resolution of each perspective image for a large view number, a super‐high‐resolution 2‐D image source is required to achieve 3‐D image quality close to the standard of natural vision. This paper proposes an approach by tiling multiple projection images with a low magnification ratio from a microdisplay to resolve the resolution issue. Placing a lenticular array in front of the tiled projection image can lead to an autostereoscopic display. Image distortion and cross‐talk issues resulting from the projection lens and pixel structure of the microdisplay have been addressed with proper selection of the active pixel and adequate pixel grouping and masking. Optical simulation has shown that a 37‐in. 12‐view autostereoscopic display with a full‐HD (1920 × 1080) resolution can be achieved with the proposed 3‐D architecture.     

The Amsterdam Library of Object Images

We present the ALOI collection of 1,000 objects recorded under various imaging circumstances. In order to capture the sensory variation in object recordings, we systematically varied viewing angle, illumination angle, and illumination color for each object, and additionally captured wide-baseline stereo images. We recorded over a hundred images of each object, yielding a total of 110,250 images for the collection. These images are made publicly available for scientific research purposes.     

Image capture pattern optimization for panoramic photography

Panoramic photography requires intensive operations of image stitching. A large quantity of images may lead to a rather expensive image stitching; while a sparse imaging may cause a poor-quality panorama due to the insufficient correlation between adjacent images. So, a good study for the balance between image quantity and image correlation may improve the efficiency and quality of panoramic photography. Therefore, in this work, we are motivated to present a novel approach to estimate the optimal image capture patterns for panoramic photography. We aim at the minimization of the image quantity which still preserves sufficient image correlation. We represent the image correlation as overlap area between the view range that can be separately observed from adjacent images. Moreover, a time-consuming imaging process of panoramic photography will result in a considerable illumination variation of the scene in images. Subsequently, the image stitching will be more challenged. To solve this problem, we design a series of imaging routines for our image capture patterns to preserve the content consistency, ensuring the generalization of our method to various cameras. Experimental results show that the proposed method can obtain the optimal image capture pattern in a very efficient manner. In these patterns, we can obtain a balanced image quantity but still achieve good results of panoramic photography.