The Visual Computing of Projector‐Camera Systems

This article focuses on real‐time image correction techniques that enable projector‐camera systems to display images onto screens that are not optimized for projections, such as geometrically complex, coloured and textured surfaces. It reviews hardware‐accelerated methods like pixel‐precise geometric warping, radiometric compensation, multi‐focal projection and the correction of general light modulation effects. Online and offline calibration as well as invisible coding methods are explained. Novel attempts in super‐resolution, high‐dynamic range and high‐speed projection are discussed. These techniques open a variety of new applications for projection displays. Some of them will also be presented in this report.     

针对多投影仪显示墙画面校正的图像对准技术

针对多投影仪显示墙画面校正问题中相机与投影仪间图像对准问题,提出一种基于自适应细分网格的稀疏对应点加密方法,可以得到任意光滑屏幕上投影图像与相机图像间的像素级对应关系。该方法利用改进的自适应4点插值细分曲线思想构造细分网格方法,可以将投影仪图像与相机图像间稀疏对应点网格加密到任意精度,从而建立从相机到投影仪图像间的像素级一一映射关系,为多投影仪显示墙系统的几何和色彩校正提供精确的图像对准基础。与现有算法的对比分析和虚拟机场塔台仿真系统中的实际应用表明,该方法具有较高的图像对准精度,并且无显式的需求投影仪、投影屏幕和相机的内部参数。     

Display adaptive 3D content remapping

Glasses-free automultiscopic displays are on the verge of becoming a standard technology in consumer products. These displays are capable of producing the illusion of 3D content without the need of any additional eyewear. However, due to limitations in angular resolution, they can only show a limited depth of field, which translates into blurred-out areas whenever an object extrudes beyond a certain depth. Moreover, the blurring is device-specific, due to the different constraints of each display. We introduce a novel display-adaptive light field retargeting method, to provide high-quality, blur-free viewing experiences of the same content on a variety of display types, ranging from hand-held devices to movie theaters. We pose the problem as an optimization, which aims at modifying the original light field so that the displayed content appears sharp while preserving the original perception of depth. In particular, we run the optimization on the central view and use warping to synthesize the rest of the light field. We validate our method using existing objective metrics for both image quality (blur) and perceived depth. The proposed framework can also be applied to retargeting disparities in stereoscopic image displays, supporting both dichotomous and non-dichotomous comfort zones.     

Modeling Color Properties of Tiled Displays

The concept of tiled displays can be successful only if such displays are made to look like a single display perceptually. The two issues that need to be solved to achieve this goal are geometric correction and color seamlessness of images spanning across tiles. Geometric correction algorithms borrow pin‐hole camera models to model projector display geometry. In this paper, we introduce an abstract modeling function that describes the color seen by a viewer when displayed by a display device. Though this function can be used to model color displayed by any common display device, in this paper, we use it to model color in multiprojector display systems. We use the model to explain the reasons for different types of color variations in a multiprojector display, to compare different color correction algorithms, and to derive such algorithms directly from the model.     

Display grid: Ad‐hoc clusters of autonomous projectors

Abstract— Projectors, like computers, are becoming commoditized. Self‐contained computers are now being networked to create computing grids, allowing transparent access to a large computing resource or massive data storage. Image presentation devices can be similarly modified to support the concept of a “display grid” to create large seamless displays. Limiting ourselves to projector‐based display grids, we present techniques for creating multi‐projector displays via self‐configuring clusters of autonomous projectors. The ad‐hoc clustering approach avoids large monolithic installations. We show a low‐cost system that supports dynamic inclusion of new projectors, automatic geometric configuration, and seamless blending of overlapping projectors.     

Automatic Registration of Multi‐Projector Domes Using a Single Uncalibrated Camera

In this paper we present a novel technique for easily calibrating multiple casually aligned projectors on spherical domes using a single uncalibrated camera. Using the prior knowledge of the display surface being a dome, we can estimate the camera intrinsic and extrinsic parameters and the projector to display surface correspondences automatically using a set of images. These images include the image of the dome itself and a projected pattern from each projector. Using these correspondences we can register images from the multiple projectors on the dome. Further, we can register displays which are not entirely visible in a single camera view using multiple pan and tilted views of an uncalibrated camera making our method suitable for displays of different size and resolution. We can register images from any arbitrary viewpoint making it appropriate for a single head‐tracked user in a 3D visualization system. Also, we can use several cartographic mapping techniques to register images in a manner that is appropriate for multi‐user visualization. Domes are known to produce a tremendous sense of immersion and presence in visualization systems. Yet, till date, there exists no easy way to register multiple projectors on a dome to create a high‐resolution realistic visualizations. To the best of our knowledge, this is the first method that can achieve accurate geometric registration of multiple projectors on a dome simply and automatically using a single uncalibrated camera.     

Projection‐based head‐tracking 3‐D displays

Abstract— This paper describes the construction and operation of four 3‐D displays in which each display produces two images for each eye and thus fits into the category of projection‐based binocular stereoscopic displays. The four 3‐D displays described are pico‐projector‐based, liquid‐ crystal—on—silicon (LCOS) conventional projector‐based, 120‐Hz digital‐light‐processor (DLP) projector‐ based, and the HELIUM3D system. In the first three displays, images are produced on a direct‐view LCD whose conventional backlight is replaced with a projection illumination source that is controlled by a multi‐user head tracker; novel steering optics direct the projector output to regions referred to as exit pupils located at the viewers' eyes. In the HELIUM3D display, the image information is supplied by a horizontally scanned, fast, light valve whose output is controlled by a spatial light modulator (SLM) to direct images to the appropriate viewers' eyes. The current statu s and the multimodal potential of the HELIUM3D display are described.     

Real‐Time Human Shadow Removal in a Front Projection System

When a person is located between a display and an operating projector, a shadow is cast on the display. The shadow on the display may eliminate important visual information and therefore adversely affect the viewing experiences. There have been various attempts to remove the human shadow cast on a projection display by using multiple projectors. While previous approaches successfully removed the shadow region when a person moderately moves around or stands stationary in front of the display, there is still an afterimage effect due to the lack of consideration of the limb motion of the person. We propose a new real‐time approach to removing the shadow cast by a person who dynamically interacts with the display, making limb motions in a front projection system. The proposed method utilizes a human skeleton obtained from a depth camera to track the posture of the person which changes over time. A model that consists of spheres and conical frustums is constructed based on the skeleton information in order to represent volumetric information of the person being tracked. Our method precisely estimates the shadow region by projecting the volumetric model onto the display. In addition, employment of intensity masks that are built based on a distance field helps suppress the afterimage of the shadow that appears when the person moves abruptly. It also helps blend the projected overlapping images from different projectors and show one smoothly combined display. The experiment results verify that our approach removes the shadow of a person effectively in a front projection environment and is fast enough to achieve real‐time performance.     

Projection‐type integral 3‐D display with distortion compensation

Abstract— Our research is aimed at developing a spatial‐imaging‐type integral three‐dimensional (3‐D) display based on an integral photography method using an extremely high‐resolution projector. One problem with the projection‐type integral 3‐D display is that geometrical distortion in projected elemental images causes spatial deformation of the displayed 3‐D image. In this study, a general relationship between the geometric distortion of elemental images and the spatial deformation of reconstructed 3‐D images were analyzed. A projection‐type integral 3‐D display with a distortion compensator which corrects the geometrical distortions of projected images in real‐time have been developed. The deformation of the displayed 3‐D images was significantly reduced by the distortion compensation, and the displayed 3‐D images had a resolution of 182 (H) × 140 (V) pixels and a viewing angle of 24.5°.     

Light-field head-mounted displays reduce the visual effort: A user study

Head-mounted displays (HMDs) allow the visualization of virtual content and the change of view perspectives in a virtual reality (VR). Besides entertainment purposes, such displays also find application in augmented reality, VR training, or tele-robotic systems. The quality of visual feedback plays a key role for the interaction performance in such setups. In the last years, high-end computers and displays led to the reduction of simulator sickness regarding nausea symptoms, while new visualization technologies are required to further reduce oculomotor and disorientation symptoms. The so-called vergence–accommodation conflict (VAC) in standard stereoscopic displays prevents intense use of 3D displays, so far. The VAC describes the visual mismatch between the projected stereoscopic 3D image and the optical distance to the HMD screen. This conflict can be solved by using displays with correct focal distance. The light-field HMD of this study provides a close-to-continuous depth and high image resolution enabling a highly natural visualization. This paper presents the first user-study on the visual comfort of light-field displays with a close-to-market HMD based on complex interaction tasks. The results provide first evidence that the light-field technology brings clear benefits to the user in terms of physical use comfort, workload, and depth matching performance.     

Seamless ultrathin rear projection display

A new architecture for a thin (2‐cm depth) rear projection display is described. In order to achieve this small depth, a very high density of rear projectors is used. Three prototype displays using rear projectors on both 5‐ and 2‐cm pitch arrays are described. The displays can achieve an effective screen pixel pitch of as small as 0.5 mm, which makes this technology competitive in terms of resolution with fine pitch LED displays; however, orders of magnitude fewer LEDs are required: Each rear projector requires only one white LED and a color liquid crystal light modulator. In the three prototypes, the projector light modulators utilize 101‐cm (40 in.), 80‐cm (31.5 in.), and 60‐cm (24 in.) diagonal liquid crystal display glass. To minimize cost, no lenses are utilized for the rear projectors. An RGB LED array may augment the projector array, which provides a low resolution component of the image onto which the high resolution component is superimposed by the projector array. Edge gaps between active areas on adjacent LCD glass units are completely eliminated by the rear projection approach enabling low profile wall‐size seamless displays. Display contrast depends on rear projection screen design.     

Spatially uniform colors for projectors and tiled displays

Abstract— A major issue when setting up multi‐projector tiled displays is the spatial non‐uniformity of the color throughout the display's area. Indeed, the chromatic properties do not only vary between two different projectors, but also between different spatial locations inside the displaying area of one single projector. A new method for calibrating the colors of a tiled display is presented. An iterative algorithm to construct a correction table which makes the luminance uniform over the projected area of one single projector is presented first. This so‐called intra‐projector calibration uses a standard camera as a luminance measuring device and can be processed in parallel for all projectors. Once the color inside each projector is spatially uniform, the set of displayable colors — the color gamut — of each projector is measured. On the basis of these measurements, the goal of the inter‐projector calibration is to find an optimal gamut shared by all the projectors. Finding the optimal color gamut displayable by n projectors in time O(n) is shown, and the color conversion from one specific color gamut to the common global gamut is derived. The method of testing it on a tiled display consisting of 48 projectors with large chrominance shifts was experimentally validated.     

基于PC集群的多屏无缝拼接技术研究

该文介绍了一种基于PC集群的多屏无缝拼接技术构造大场景显示系统的方法。大面积高分辨率的投影系统在各个领域获得了越来越广泛的应用,但系统构建费用高、维护要求严。在对比分析了传统投影系统的优缺点和目前存在的解决方案后,该方法使用普通PC、数码像机和以太网构建的硬件系统,提出了基于相机的自动几伺校准和基于线性函数的边缘融合方法,消除了投影画面之间的缝隙,并采用Master／Slave结构控制图形的同步渲染,同时支持分布交互式控制。这种方法在满足同等投影显示要求的情况下。将大幅度降低了系统的构建费用和维护要求。     

Advanced MEMS-based technologies and displays

MEMS (microelectromechanical systems) are used in many fields including display applications, which are extensively studied both in academia and industry. For practical devices, numbers of advanced technologies have been developed based on MEMS concept. For display technologies, projection displays, reflective displays, transmissive displays and other display modes have been achieved by different MEMS modes. In this review, the current MEMS-based display technologies are introduced and discussed including digital micromirror device (DMD), laser scanning display (LSD), interferometric modulator display (IMOD), digital micro-shutter (DMS), time multiplexed optical shutter (TMOS), grating light valve (GLV) and others. The typical structure and fundamental of each display mode are interpreted.     

Surround structured lighting: 3-D scanning with orthographic illumination

This paper presents a new system for rapidly acquiring complete 3-D surface models using a single orthographic structured light projector, a pair of planar mirrors, and one or more synchronized cameras. Using the mirrors, we project structured light patterns that illuminate the object from all sides (not just the side of the projector) and are able to observe the object from several vantage points simultaneously. This system requires that projected planes of light to be parallel, so we construct an orthographic projector using a Fresnel lens and a commercial DLP projector. A single Gray code sequence is used to encode a set of vertically-spaced light planes within the scanning volume, and five views of the illuminated object are obtained from a single image of the planar mirrors located behind it. From each real and virtual camera we recover a dense 3-D point cloud spanning the entire object surface using traditional structured light algorithms. A key benefit of this design is to ensure that each point on the object surface can be assigned an unambiguous Gray code sequence, despite the possibility of being illuminated from multiple directions. In addition to presenting a prototype implementation, we also develop a complete set of mechanical alignment and calibration procedures for utilizing orthographic projectors in computer vision applications. As we demonstrate, the proposed system overcomes a major hurdle to achieving full 360° reconstructions using a single structured light sequence by eliminating the need for merging multiple scans or multiplexing several projectors.     

Advances towards high‐resolution pack‐and‐go displays: A survey

Abstract— Tiled displays provide high resolution and large scale simultaneously. Projectors can project on any available surface. Thus, it is possible to create a large high‐resolution display by simply tiling multiple projectors on any available regular surface. The tremendous advancement in projection technology has made projectors portable and affordable. One can envision displays made of multiple such projectors that can be packed in one's car trunk, carried from one location to another, deployed at each location easily to create a seamless high‐resolution display, and, finally, dismantled in minutes to be taken to the next location — essentially a pack‐and‐go display. Several challenges must be overcome in order to realize such pack‐and‐go displays. These include allowing for imperfect uncalibrated devices, uneven non‐diffused display surfaces, and a layman user via complete automation in deployment that requires no user invention. We described the advances we have made in addressing these challenges for the most common case of planar display surfaces. First, we present a technique to allow imperfect projectors. Next, we present a technique to allow a photometrically uncalibrated camera. Finally, we present a novel distributed architecture that renders critical display capabilities such as self‐calibration, scalability, and reconfigurability without any user intervention. These advances are important milestones towards the development of easy‐to‐use multi‐projector displays that can be deployed anywhere and by anyone.     

Visual quality evaluation of large LED displays based on subjective sensory perception

Large light-emitting diode (LED) displays have undergone a great deal of development in recent years. One of their main characteristics is their potential for great diversity in their physical parameters (size, brightness, power consumption, etc.) and in the configuration of the LED pixel arrays. The physical parameters can vary by several orders of magnitude, while the matrix can be organized in discrete real RGB pixels or in several types of virtual pixel distributions such as Bayer, Hexagonal and Diagonal. For this reason, it is particularly difficult to make a complete and exhaustive evaluation of any type of displays based only on objective parameters. In this paper we have developed and used a subjective method for evaluating the image quality of various types of LED displays. The results obtained show that there is not a perfect display solution and that for each particular project it is possible to find an optimal LED matrix solution based on quality performance and the number and distribution of LEDs used (that marks pixel resolution, overall manufacturing cost and power consumption). For displays with the same number of LEDs the array of emitters formed by real pixels are desirable for applications viewed at short distances and for predominantly alphanumeric content, while virtual pixels arrays are advantageous in applications requiring greater viewing distances and a higher content of complex images.     

Thin‐film micromirror array (TMA) for large information‐display systems

Abstract— The thin‐film micromirror array (TMA) is a new reflective‐type spatial light modulator fabricated with microelectromechanical systems (MEMS) technology. Micromachined thin‐film piezoelectric actuators are used to control the tilt angle of each micromirror, which defines the gray scale of the matching screen pixel. A working projector prototype of 5400 true ANSI lumens was presented at Asian Display '98 with three VGA‐format TMAchips and a 1‐kW xenon lamp. The prototype showed a light‐transmitting efficiency of 22%. An XGA‐format TMA was developed for commercial purposes. The size of each micromirror of the XGA‐format TMA is reduced to half the size of the micromirror of a VGA‐format TMA. Efforts have been made to maintain the light‐transmitting efficiency of the XGA TMA projector over 21%, as well as to improve the uniformity.     

Practical Radiometric Compensation for Projection Display on Textured Surfaces using a Multidimensional Model

Radiometric compensation methods remove the effect of the underlying spatially varying surface reflectance of the texture when projecting on textured surfaces. All prior work sample the surface reflectance dependent radiometric transfer function from the projector to the camera at every pixel that requires the camera to observe tens or hundreds of images projected by the projector. In this paper, we cast the radiometric compensation problem as a sampling and reconstruction of multi‐dimensional radiometric transfer function that models the color transfer function from the projector to an observing camera and the surface reflectance in a unified manner. Such a multi‐dimensional representation makes no assumption about linearity of the projector to camera color transfer function and can therefore handle projectors with non‐linear color transfer functions(e.g. DLP, LCOS, LED‐based or laser‐based). We show that with a well‐curated sampling of this multi‐dimensional function, achieved by exploiting the following key properties, is adequate for its accurate representation: (a) the spectral reflectance of most real‐world materials are smooth and can be well‐represented using a lower‐dimension function; (b) the reflectance properties of the underlying texture have strong redundancies – for example, multiple pixels or even regions can have similar surface reflectance; (c) the color transfer function from the projector to camera have strong input coherence. The proposed sampling allows us to reduce the number of projected images that needs to be observed by a camera by up to two orders of magnitude, the minimum being only two. We then present a new multi‐dimensional scattered data interpolation technique to reconstruct the radiometric transfer function at a high spatial density (i.e. at every pixel) to compute the compensation image. We show that the accuracy of our interpolation technique is higher than any existing methods.     

面向球幕投影系统的几何校正方法

摘 要：用于飞行模拟的视景仿真系统，经常会以多台投影仪同步投影以得到较大范围的 视场角。当投影机斜对屏幕或者投影屏幕为曲面时，图像会发生几何失真。针对此问题，本文 提出了一种专门面向球幕投影系统的几何校正方法，并以一个三通道显示系统为实例，详述了 该方法的理论原理以及校正流程。实例结果表明，经几何校正后，各投影图像无几何畸变，通 道过渡处几何内容完全一致。该方法是一种纯软件方法，成本小而且操作简单，能够适应于不 同的投影场景。