Scalable Multi‐view Registration for Multi‐Projector Displays on Vertically Extruded Surfaces

Recent work have shown that it is possible to register multiple projectors on non‐planar surfaces using a single uncalibrated camera instead of a calibrated stereo pair when dealing with a special class of non‐planar surfaces, vertically extruded surfaces. However, this requires the camera view to contain the entire display surface. This is often an impossible scenario for large displays, especially common in visualization, edutainment, training and simulation applications. In this paper we present a new method that can achieve an accurate geometric registration even when the field‐of‐view of the uncalibrated camera can cover only a part of the vertically extruded display at a time. We pan and tilt the camera from a single point and employ a multi‐view approach to register the projectors on the display. This allows the method to scale easily both in terms of camera resolution and display size. To the best of our knowledge, our method is the first to achieve a scalable multi‐view geometric registration of large vertically extruded displays with a single uncalibrated camera. This method can also handle a different situation of having multiple similarly oriented cameras in different locations, if the camera focal length is known.     

Advances towards high‐resolution pack‐and‐go displays: A survey

Abstract— Tiled displays provide high resolution and large scale simultaneously. Projectors can project on any available surface. Thus, it is possible to create a large high‐resolution display by simply tiling multiple projectors on any available regular surface. The tremendous advancement in projection technology has made projectors portable and affordable. One can envision displays made of multiple such projectors that can be packed in one's car trunk, carried from one location to another, deployed at each location easily to create a seamless high‐resolution display, and, finally, dismantled in minutes to be taken to the next location — essentially a pack‐and‐go display. Several challenges must be overcome in order to realize such pack‐and‐go displays. These include allowing for imperfect uncalibrated devices, uneven non‐diffused display surfaces, and a layman user via complete automation in deployment that requires no user invention. We described the advances we have made in addressing these challenges for the most common case of planar display surfaces. First, we present a technique to allow imperfect projectors. Next, we present a technique to allow a photometrically uncalibrated camera. Finally, we present a novel distributed architecture that renders critical display capabilities such as self‐calibration, scalability, and reconfigurability without any user intervention. These advances are important milestones towards the development of easy‐to‐use multi‐projector displays that can be deployed anywhere and by anyone.     

Registration techniques for using imperfect and partially calibrated devices in planar multi-projector displays

Multi-projector displays today are automatically registered, both geometrically and photometrically, using cameras. Existing registration techniques assume pre-calibrated projectors and cameras that are devoid of imperfections such as lens distortion. In practice, however, these devices are usually imperfect and uncalibrated. Registration of each of these devices is often more challenging than the multi-projector display registration itself. To make tiled projection-based displays accessible to a layman user we should allow the use of uncalibrated inexpensive devices that are prone to imperfections. In this paper, we make two important advances in this direction. First, we present a new geometric registration technique that can achieve geometric alignment {\em in the presence of severe projector lens distortion} using a relatively inexpensive low-resolution camera. This is achieved via a closed-form model that relates the projectors to cameras, in planar multi-projector displays, using rational Bezier patches. This enables us to geometrically calibrate a 3000 x 2500 resolution planar multi-projector display made of 3 x 3 array of nine severely distorted projectors using a low resolution (640 x 480) VGA camera. Second, we present a photometric self-calibration technique for a projector-camera pair. This allows us to photometrically calibrate the same display made of nine projectors using a photometrically uncalibrated camera. To the best of our knowledge, this is the first work that allows geometrically imperfect projectors and photometrically uncalibrated cameras in calibrating multi-projector displays.     

Autocalibration of multiprojector CAVE-like immersive environments

In this paper, we present the first method for the geometric autocalibration of multiple projectors on a set of CAVE-like immersive display surfaces including truncated domes and 4 or 5-wall CAVEs (three side walls, floor, and/or ceiling). All such surfaces can be categorized as swept surfaces and multiple projectors can be registered on them using a single uncalibrated camera without using any physical markers on the surface. Our method can also handle nonlinear distortion in the projectors, common in compact setups where a short throw lens is mounted on each projector. Further, when the whole swept surface is not visible from a single camera view, we can register the projectors using multiple pan and tilted views of the same camera. Thus, our method scales well with different size and resolution of the display. Since we recover the 3D shape of the display, we can achieve registration that is correct from any arbitrary viewpoint appropriate for head-tracked single-user virtual reality systems. We can also achieve wallpapered registration, more appropriate for multiuser collaborative explorations. Though much more immersive than common surfaces like planes and cylinders, general swept surfaces are used today only for niche display environments. Even the more popular 4 or 5-wall CAVE is treated as a piecewise planar surface for calibration purposes and hence projectors are not allowed to be overlapped across the corners. Our method opens up the possibility of using such swept surfaces to create more immersive VR systems without compromising the simplicity of having a completely automatic calibration technique. Such calibration allows completely arbitrary positioning of the projectors in a 5-wall CAVE, without respecting the corners.     

Asynchronous distributed calibration for scalable and reconfigurable multi-projector displays

Centralized techniques have been used until now when automatically calibrating (both geometrically and photometrically) large high-resolution displays created by tiling multiple projectors in a 2D array. A centralized server managed all the projectors and also the camera(s) used to calibrate the display. In this paper, we propose an asynchronous distributed calibration methodology via a display unit called the plug-and-play projector (PPP). The PPP consists of a projector, camera, computation and communication unit, thus creating a self-sufficient module that enables an asynchronous distributed architecture for multi-projector displays. We present a single-program-multiple-data (SPMD) calibration algorithm that runs on each PPP and achieves a truly scalable and reconfigurable display without any input from the user. It instruments novel capabilities like adding/removing PPPs from the display dynamically, detecting faults, and reshaping the display to a reasonable rectangular shape to react to the addition/removal/faults. To the best of our knowledge, this is the first attempt to realize a completely asynchronous and distributed calibration architecture and methodology for multi-projector displays.     

Easy Calibration of a Multi-projector Display System

In this paper, we present a method for the geometric calibration of a multi-projector display system. The method is such that in order to calibrate the system, the user is only required to place the projectors and capture a single image of the images projected from them onto a planar screen using a hand-held camera. The problem to be solved is divided into the image registration for stitching different projector images into a single seamless image and the image rectification for making the image have the correct rectangular shape. The proposed method is characterized by simultaneously solving both of them from only a single image, which makes the calibration procedures easy. The method assumes an uncalibrated camera and partially calibrated projectors in which only focal lengths are unknown among the internal parameters. In the paper, we first prove the uniqueness of solutions to the problem, which was unclear in the previous studies, and then present a stable numerical algorithm for actually finding the solution. We present several experimental results for synthetic data, in which we show the relation between the calibration accuracy and several factors, and also present experimental results for real data, in which we demonstrate that the proposed method can calibrate a real system with sufficient accuracy for a number of layouts of the projectors.     

Introduction to building projection-based tiled display systems

This tutorial introduces the concepts and technologies needed to build projector-based display systems. Tiled displays offer scalability, high resolution, and large formats for various applications. Tiled displays are an emerging technology for constructing semi-immersive visualization environments capable of presenting high-resolution images from scientific simulation. The largest impact may well arise from using large-format tiled displays as one of possibly multiple displays in building information or active spaces that surround the user with diverse ways of interacting with data and multimedia information flows. These environments may prove the ultimate successor to the desktop metaphor for information technology work. Several fundamental technological problems must be addressed to make tiled displays practical. These include: the choice of screen materials and support structures; choice of projectors, projector supports, and optional fine positioners; techniques for integrating image tiles into a seamless whole; interface devices for interaction with applications; display generators and interfaces; and the display software environment     

A scalable distributed paradigm for multi-user interaction with tiled rear projection display walls

We present the first distributed paradigm for multiple users to interact simultaneously with large tiled rear projection display walls. Unlike earlier works, our paradigm allows easy scalability across different applications, interaction modalities, displays and users. The novelty of the design lies in its distributed nature allowing well-compartmented, application independent, and application specific modules. This enables adapting to different 2D applications and interaction modalities easily by changing a few application specific modules. We demonstrate four challenging 2D applications on a nine projector display to demonstrate the application scalability of our method: map visualization, virtual graffiti, virtual bulletin board and an emergency management system. We demonstrate the scalability of our method to multiple interaction modalities by showing both gesture-based and laser-based user interfaces. Finally, we improve earlier distributed methods to register multiple projectors. Previous works need multiple patterns to identify the neighbors, the configuration of the display and the registration across multiple projectors in logarithmic time with respect to the number of projectors in the display. We propose a new approach that achieves this using a single pattern based on specially augmented QR codes in constant time. Further, previous distributed registration algorithms are prone to large misregistrations. We propose a novel radially cascading geometric registration technique that yields significantly better accuracy. Thus, our improvements allow a significantly more efficient and accurate technique for distributed self-registration of multi-projector display walls.     

A unified paradigm for scalable multi-projector displays

We present a general framework for the modeling and optimization of scalable multi-projector displays. Based on this framework, we derive algorithms that can robustly optimize the visual quality of an arbitrary combination of projectors without manual adjustment. When the projectors are tiled, we show that our framework automatically produces blending maps that outperform state-of-the-art projector blending methods. When all the projectors are superimposed, the framework can produce high-resolution images beyond the Nyquist resolution limits of component projectors. When a combination of tiled and superimposed projectors are deployed, the same framework harnesses the best features of both tiled and superimposed multi-projector projection paradigms. The framework creates for the first time a new unified paradigm that is agnostic to a particular configuration of projectors yet robustly optimizes for the brightness, contrast, and resolution of that configuration. In addition, we demonstrate that our algorithms support high resolution video at real-time interactive frame rates achieved on commodity graphics platforms. This work allows for inexpensive, compelling, flexible, and robust large scale visualization systems to be built and deployed very efficiently.     

Seamless ultrathin rear projection display

A new architecture for a thin (2‐cm depth) rear projection display is described. In order to achieve this small depth, a very high density of rear projectors is used. Three prototype displays using rear projectors on both 5‐ and 2‐cm pitch arrays are described. The displays can achieve an effective screen pixel pitch of as small as 0.5 mm, which makes this technology competitive in terms of resolution with fine pitch LED displays; however, orders of magnitude fewer LEDs are required: Each rear projector requires only one white LED and a color liquid crystal light modulator. In the three prototypes, the projector light modulators utilize 101‐cm (40 in.), 80‐cm (31.5 in.), and 60‐cm (24 in.) diagonal liquid crystal display glass. To minimize cost, no lenses are utilized for the rear projectors. An RGB LED array may augment the projector array, which provides a low resolution component of the image onto which the high resolution component is superimposed by the projector array. Edge gaps between active areas on adjacent LCD glass units are completely eliminated by the rear projection approach enabling low profile wall‐size seamless displays. Display contrast depends on rear projection screen design.     

Projector placement planning for high quality visualizations on real-world colored objects

Many visualization applications benefit from displaying content on real-world objects rather than on a traditional display (e.g., a monitor). This type of visualization display is achieved by projecting precisely controlled illumination from multiple projectors onto the real-world colored objects. For such a task, the placement of the projectors is critical in assuring that the desired visualization is possible. Using ad hoc projector placement may cause some appearances to suffer from color shifting due to insufficient projector light radiance being exposed onto the physical surface. This leads to an incorrect appearance and ultimately to a false and potentially misleading visualization. In this paper, we present a framework to discover the optimal position and orientation of the projectors for such projection-based visualization displays. An optimal projector placement should be able to achieve the desired visualization with minimal projector light radiance. When determining optimal projector placement, object visibility, surface reflectance properties, and projector-surface distance and orientation need to be considered. We first formalize a theory for appearance editing image formation and construct a constrained linear system of equations that express when a desired novel appearance or visualization is possible given a geometric and surface reflectance model of the physical surface. Then, we show how to apply this constrained system in an adaptive search to efficiently discover the optimal projector placement which achieves the desired appearance. Constraints can be imposed on the maximum radiance allowed by the projectors and the projectors' placement to support specific goals of various visualization applications. We perform several real-world and simulated appearance edits and visualizations to demonstrate the improvement obtained by our discovered projector placement over ad hoc projector placement.     

Spatially uniform colors for projectors and tiled displays

Abstract— A major issue when setting up multi‐projector tiled displays is the spatial non‐uniformity of the color throughout the display's area. Indeed, the chromatic properties do not only vary between two different projectors, but also between different spatial locations inside the displaying area of one single projector. A new method for calibrating the colors of a tiled display is presented. An iterative algorithm to construct a correction table which makes the luminance uniform over the projected area of one single projector is presented first. This so‐called intra‐projector calibration uses a standard camera as a luminance measuring device and can be processed in parallel for all projectors. Once the color inside each projector is spatially uniform, the set of displayable colors — the color gamut — of each projector is measured. On the basis of these measurements, the goal of the inter‐projector calibration is to find an optimal gamut shared by all the projectors. Finding the optimal color gamut displayable by n projectors in time O(n) is shown, and the color conversion from one specific color gamut to the common global gamut is derived. The method of testing it on a tiled display consisting of 48 projectors with large chrominance shifts was experimentally validated.     

基于图像序列的交互式快速建模系统

给出了一个基于图像序列的交互式三维建模系统.通过输入一段未标定的图像或视频序列,系统能够自动地恢复出摄像机参数;然后用户只需要在少量几帧图像上简单勾画出物体的形态结构,系统就能自动解析出多帧之间用户交互的对应关系,从而迅速、逼真地重建出场景的三维模型.该系统提供了点与线段的重建、直线与平面的重建、曲线与曲面的重建等功能,能够满足对现实世界中的复杂场景的快速高精度的重建要求.几组真实拍摄的图像序列的建模实验表明:该系统高效、实用.能够很好地满足实际建模需求.     

Head tracked retroreflecting 3D display

In this paper, we describe a single‐user glasses‐free (autostereoscopic) 3D display where images from a pair of picoprojectors are projected on to a retroreflecting screen. Real images of the projector lenses formed at the viewer's eyes produce exit pupils that follow the eye positions by the projectors moving laterally under the control of a head tracker. This provides the viewer with a comfortable degree of head movement. The retroreflecting screen, display hardware, infrared head tracker, and means of stabilizing the image position on the screen are explained. The performance of the display in terms of crosstalk, resolution, image distortion, and other parameters is described. Finally, applications of this display type are suggested.     

Display grid: Ad‐hoc clusters of autonomous projectors

Abstract— Projectors, like computers, are becoming commoditized. Self‐contained computers are now being networked to create computing grids, allowing transparent access to a large computing resource or massive data storage. Image presentation devices can be similarly modified to support the concept of a “display grid” to create large seamless displays. Limiting ourselves to projector‐based display grids, we present techniques for creating multi‐projector displays via self‐configuring clusters of autonomous projectors. The ad‐hoc clustering approach avoids large monolithic installations. We show a low‐cost system that supports dynamic inclusion of new projectors, automatic geometric configuration, and seamless blending of overlapping projectors.     

Color and brightness appearance issues in tiled displays

Large-format displays created by tiling multiple, projected images have been used for decades in flight simulators and entertainment and are commercially available in a variety of forms. More recently, various research organizations have built custom display walls out of commodity projectors to support research in visualization, large-format display, and interaction. In these settings, making the display appear as a single, seamless surface has proven challenging. Where tiles overlap, they create bright seams. The tiles vary in color and brightness, not only from tile to tile, but within each tile. Each projector has a slightly different color gamut, caused by variations in the bulb, color filters, and digital processing (contrast, brightness, and gamma) for the projector. The spatial variation in brightness has two causes. First, the light from a projection system doesn't uniformly illuminate the screen. Second, the light doesn't scatter uniformly out of the front of the screen, making the perceived brightness depend on the viewing angle. In some projectors, the projected light's color also varies across the tile's face, resulting in unwanted tints in the images. I describe what causes these variations and what can be done about them     

Modeling Color Properties of Tiled Displays

The concept of tiled displays can be successful only if such displays are made to look like a single display perceptually. The two issues that need to be solved to achieve this goal are geometric correction and color seamlessness of images spanning across tiles. Geometric correction algorithms borrow pin‐hole camera models to model projector display geometry. In this paper, we introduce an abstract modeling function that describes the color seen by a viewer when displayed by a display device. Though this function can be used to model color displayed by any common display device, in this paper, we use it to model color in multiprojector display systems. We use the model to explain the reasons for different types of color variations in a multiprojector display, to compare different color correction algorithms, and to derive such algorithms directly from the model.     

Color nonuniformity in projection-based displays: analysis and solutions

Large-area displays made up of several projectors show significant variation in color. Here, we identify different projector parameters that cause the color variation and study their effects on the luminance and chrominance characteristics of the display. This work leads to the realization that luminance varies significantly within and across projectors, while chrominance variation is relatively small, especially across projectors of same model. To address this situation, we present a method to achieve luminance matching across all pixels of a multiprojector display that results in photometrically uniform displays. We use a camera as a measurement device for this purpose. Our method comprises a one-time calibration step that generates a per channel per projector luminance attenuation map (LAM), which is then used to correct any image projected on the display at interactive rates on commodity graphics hardware. To the best of our knowledge, this is the first effort to match luminance across all the pixels of a multiprojector display.     

A high‐resolution autostereoscopic display system with a wide viewing angle using an LCOS projector array

Abstract— This study develops an autostereoscopic display based on a multiple miniature projector array to provide a scalable solution for a high‐resolution 3‐D display with large viewing freedom. To minimize distortion and dispersion, and to maximize the modulation transfer function (MTF) of the projection image to optimize 3‐D image quality, a dedicated projection lens and an accurate six‐axis adjusting platform for the miniature projector were designed and fabricated. Image‐blending technology based on a lookup table was adopted to combine images from 30 miniature projectors into a seamless single image. The result was a 35‐in. autostereoscopic display with 60 views ata 30° viewing angle, 90° FOV, and large range of viewing distance. The proposed system exhibits very smooth motion parallax when viewers move around in front of it.