Camera-based calibration techniques for seamless multiprojector displays

Multiprojector, large-scale displays are used in scientific visualization, virtual reality, and other visually intensive applications. In recent years, a number of camera-based computer vision techniques have been proposed to register the geometry and color of tiled projection-based display. These automated techniques use cameras to "calibrate" display geometry and photometry, computing per-projector corrective warps and intensity corrections that are necessary to produce seamless imagery across projector mosaics. These techniques replace the traditional labor-intensive manual alignment and maintenance steps, making such displays cost-effective, flexible, and accessible. In this paper, we present a survey of different camera-based geometric and photometric registration techniques reported in the literature to date. We discuss several techniques that have been proposed and demonstrated, each addressing particular display configurations and modes of operation. We overview each of these approaches and discuss their advantages and disadvantages. We examine techniques that address registration on both planar (video walls) and arbitrary display surfaces and photometric correction for different kinds of display surfaces. We conclude with a discussion of the remaining challenges and research opportunities for multiprojector displays     

基于单投影仪与柱面反射镜的沉浸感显示系统

在分析传统沉浸感显示系统优缺点的基础上,设计并实现了一种新型的沉浸感显示系统。该系统使用柱面反射镜对单投影仪投射光线进行反射,并在弧形背投幕上成像,能够获得无缝画面,实现了广角度虚拟场景的连贯显示。通过合理设计柱面反射镜形状,实现了投影画面在水平方向上的均匀放大。通过对图像进行预变形可以基本消除因投影幕曲率而产生的投影画面形变,同时采用背投技术使参与者在虚拟场景中活动更加自如。该系统克服了传统的多投影仪或多显示器沉浸感显示系统中存在的图像拼接问题,且易于构建,经实验验证能够获得良好的沉浸感显示效果。     

Modeling Color Properties of Tiled Displays

The concept of tiled displays can be successful only if such displays are made to look like a single display perceptually. The two issues that need to be solved to achieve this goal are geometric correction and color seamlessness of images spanning across tiles. Geometric correction algorithms borrow pin‐hole camera models to model projector display geometry. In this paper, we introduce an abstract modeling function that describes the color seen by a viewer when displayed by a display device. Though this function can be used to model color displayed by any common display device, in this paper, we use it to model color in multiprojector display systems. We use the model to explain the reasons for different types of color variations in a multiprojector display, to compare different color correction algorithms, and to derive such algorithms directly from the model.     

Color nonuniformity in projection-based displays: analysis and solutions

Large-area displays made up of several projectors show significant variation in color. Here, we identify different projector parameters that cause the color variation and study their effects on the luminance and chrominance characteristics of the display. This work leads to the realization that luminance varies significantly within and across projectors, while chrominance variation is relatively small, especially across projectors of same model. To address this situation, we present a method to achieve luminance matching across all pixels of a multiprojector display that results in photometrically uniform displays. We use a camera as a measurement device for this purpose. Our method comprises a one-time calibration step that generates a per channel per projector luminance attenuation map (LAM), which is then used to correct any image projected on the display at interactive rates on commodity graphics hardware. To the best of our knowledge, this is the first effort to match luminance across all the pixels of a multiprojector display.     

Connected without disconnection: Overview of light field metaverse applications and their quality of experience

With the rapid technological advances of the recent years, the practical instances of the metaverse have become more immersive and engaging than ever before. In their most frequent forms, the 3D virtual worlds of the metaverse are enabled by virtual reality headsets. This means that the user is completely disconnected from its real environment and is fully immersed in such a virtual world. The quality of experience of virtual reality and other headset-based technologies is now definitely a hot research topic, and the findings of the relevant scientific efforts are continuously emerging. However, as a headset-free immersive 3D technology, light field visualization is greatly underinvestigated with regard to the concept of the metaverse. In this paper, we address the applications of light field metaverse, compare its advantages and disadvantages to more conventional metaverse technologies, and discuss the most important issues regarding user experience. The paper highlights the user-oriented considerations for the development of general-purpose and dedicated light field displays. Additionally, our work examines state-of-the-art display systems and the current feasibility of a light field metaverse.     

Egocentric distance perception in large screen immersive displays

Many scientists have demonstrated that compared to the real world egocentric distances in head-mounted display virtual environments are underestimated. However, distance perception in large screen immersive displays has received less attention. We investigate egocentric distance perception in a virtual office room projected using a semi-spherical, a Max Planck Institute CyberMotion Simulator cabin and a flat large screen immersive display. The goal of our research is to systematically investigate distance perception in large screen immersive displays with commonly used technical specifications. We specifically investigate the role of distance to the target, stereoscopic projection and motion parallax on distance perception. We use verbal reports and blind walking as response measures for the real world experiment. Due to the limited space in the three large screen immersive displays we use only verbal reports as the response measure for the experiments in the virtual environment. Our results show an overall underestimation of distance perception in the large screen immersive displays, while verbal estimates of distances are nearly veridical in the real world. We find that even when providing motion parallax and stereoscopic depth cues to the observer in the flat large screen immersive display, participants estimate the distances to be smaller than intended. Although stereo cues in the flat large screen immersive display do increase distance estimates for the nearest distance, the impact of the stereoscopic depth cues is not enough to result in veridical distance perception. Further, we demonstrate that the distance to the target significantly influences the percent error of verbal estimates in both the real and virtual world. The impact of the distance to the target on the distance judgments is the same in the real world and in two of the used large screen displays, namely, the MPI CyberMotion Simulator cabin and the flat displays. However, in the semi-spherical display we observe a significantly different influence of distance to the target on verbal estimates of egocentric distances. Finally, we discuss potential reasons for our results. Based on the findings from our research we give general suggestions that could serve as methods for improving the LSIDs in terms of the accuracy of depth perception and suggest methods to compensate for the underestimation of verbal distance estimates in large screen immersive displays.     

A scalable architecture for geometric correction of multi-projector display systems

Multi-projector displays allow the realization of large and immersive projection environments by allowing the tiling of projections from multiple projectors. Such tiled displays require real time geometrical warping of the content that is being projected from each projector. This geometrical warping is a computationally intensive operation and is typically applied using high-end graphics processing units (GPUs) that are able to process a defined number of projector channels. Furthermore, this limits the applicability of such multi-projector display systems only to the content that is being generated using desktop based systems. In this paper we propose a platform independent FPGA based scalable hardware architecture for geometric correction of projected content that allows addition of each projector channel at a fractional increase in logic area. The proposed scheme provides real time correction of HD quality video streams and thus enables the use of this technology for embedded and standalone devices.     

Introduction to building projection-based tiled display systems

This tutorial introduces the concepts and technologies needed to build projector-based display systems. Tiled displays offer scalability, high resolution, and large formats for various applications. Tiled displays are an emerging technology for constructing semi-immersive visualization environments capable of presenting high-resolution images from scientific simulation. The largest impact may well arise from using large-format tiled displays as one of possibly multiple displays in building information or active spaces that surround the user with diverse ways of interacting with data and multimedia information flows. These environments may prove the ultimate successor to the desktop metaphor for information technology work. Several fundamental technological problems must be addressed to make tiled displays practical. These include: the choice of screen materials and support structures; choice of projectors, projector supports, and optional fine positioners; techniques for integrating image tiles into a seamless whole; interface devices for interaction with applications; display generators and interfaces; and the display software environment     

Projector placement planning for high quality visualizations on real-world colored objects

Many visualization applications benefit from displaying content on real-world objects rather than on a traditional display (e.g., a monitor). This type of visualization display is achieved by projecting precisely controlled illumination from multiple projectors onto the real-world colored objects. For such a task, the placement of the projectors is critical in assuring that the desired visualization is possible. Using ad hoc projector placement may cause some appearances to suffer from color shifting due to insufficient projector light radiance being exposed onto the physical surface. This leads to an incorrect appearance and ultimately to a false and potentially misleading visualization. In this paper, we present a framework to discover the optimal position and orientation of the projectors for such projection-based visualization displays. An optimal projector placement should be able to achieve the desired visualization with minimal projector light radiance. When determining optimal projector placement, object visibility, surface reflectance properties, and projector-surface distance and orientation need to be considered. We first formalize a theory for appearance editing image formation and construct a constrained linear system of equations that express when a desired novel appearance or visualization is possible given a geometric and surface reflectance model of the physical surface. Then, we show how to apply this constrained system in an adaptive search to efficiently discover the optimal projector placement which achieves the desired appearance. Constraints can be imposed on the maximum radiance allowed by the projectors and the projectors' placement to support specific goals of various visualization applications. We perform several real-world and simulated appearance edits and visualizations to demonstrate the improvement obtained by our discovered projector placement over ad hoc projector placement.     

Automatic Registration of Multi‐Projector Domes Using a Single Uncalibrated Camera

In this paper we present a novel technique for easily calibrating multiple casually aligned projectors on spherical domes using a single uncalibrated camera. Using the prior knowledge of the display surface being a dome, we can estimate the camera intrinsic and extrinsic parameters and the projector to display surface correspondences automatically using a set of images. These images include the image of the dome itself and a projected pattern from each projector. Using these correspondences we can register images from the multiple projectors on the dome. Further, we can register displays which are not entirely visible in a single camera view using multiple pan and tilted views of an uncalibrated camera making our method suitable for displays of different size and resolution. We can register images from any arbitrary viewpoint making it appropriate for a single head‐tracked user in a 3D visualization system. Also, we can use several cartographic mapping techniques to register images in a manner that is appropriate for multi‐user visualization. Domes are known to produce a tremendous sense of immersion and presence in visualization systems. Yet, till date, there exists no easy way to register multiple projectors on a dome to create a high‐resolution realistic visualizations. To the best of our knowledge, this is the first method that can achieve accurate geometric registration of multiple projectors on a dome simply and automatically using a single uncalibrated camera.     

3DIVE: An Immersive Environment for Interactive Volume Data Exploration

This paper describes an immersive system,called 3DIVE,for interactive volume data visualization and exploration inside the CAVE virtual environment.Combining interactive volume rendering and virtual reality provides a netural immersive environment for volumetric data visualization.More advanced data exploration operations,such as object level data manipulation,simulation and analysis ,are supported in 3DIVE by several new techniques,In particular,volume primitives and texture regions ae used for the rendering,manipulation,and collision detection of volumetric objects;and the region-based rendering pipeline is integrated with 3D image filters to provide an image-based mechanism for interactive transfer function design.The system has been recently released as public domain software for CAVE/ImmersaDesk users,and is currently being actively used by various scientific and biomedical visualization projects.     

Human factors in the design of an immersive display

In 1988 Fakespace began building a telepresence camera system for the Virtual Environment Workstation (View) project at NASA Ames Research Center. The complete system combined a teleoperated camera platform and 3D viewing system. Fakespace has installed descendents of this system for day-to-day use in environments ranging from research laboratories to office cubicles. By observing how people use image-generation and viewing technologies, we have evaluated the human factors involved in building and deploying effective immersive visualization systems. This article describes the application of these factors in the design of the Fakespace BOOM (Binocular Omni-Orientation Monitor). The Fakespace BOOM represents a class of immersive display devices known as counterbalanced displays. The choices that led to the design of this device apply to the design of many types of immersive displays     

Autocalibration of multiprojector CAVE-like immersive environments

In this paper, we present the first method for the geometric autocalibration of multiple projectors on a set of CAVE-like immersive display surfaces including truncated domes and 4 or 5-wall CAVEs (three side walls, floor, and/or ceiling). All such surfaces can be categorized as swept surfaces and multiple projectors can be registered on them using a single uncalibrated camera without using any physical markers on the surface. Our method can also handle nonlinear distortion in the projectors, common in compact setups where a short throw lens is mounted on each projector. Further, when the whole swept surface is not visible from a single camera view, we can register the projectors using multiple pan and tilted views of the same camera. Thus, our method scales well with different size and resolution of the display. Since we recover the 3D shape of the display, we can achieve registration that is correct from any arbitrary viewpoint appropriate for head-tracked single-user virtual reality systems. We can also achieve wallpapered registration, more appropriate for multiuser collaborative explorations. Though much more immersive than common surfaces like planes and cylinders, general swept surfaces are used today only for niche display environments. Even the more popular 4 or 5-wall CAVE is treated as a piecewise planar surface for calibration purposes and hence projectors are not allowed to be overlapped across the corners. Our method opens up the possibility of using such swept surfaces to create more immersive VR systems without compromising the simplicity of having a completely automatic calibration technique. Such calibration allows completely arbitrary positioning of the projectors in a 5-wall CAVE, without respecting the corners.     

基于PC集群的多屏无缝拼接技术研究

该文介绍了一种基于PC集群的多屏无缝拼接技术构造大场景显示系统的方法。大面积高分辨率的投影系统在各个领域获得了越来越广泛的应用,但系统构建费用高、维护要求严。在对比分析了传统投影系统的优缺点和目前存在的解决方案后,该方法使用普通PC、数码像机和以太网构建的硬件系统,提出了基于相机的自动几伺校准和基于线性函数的边缘融合方法,消除了投影画面之间的缝隙,并采用Master／Slave结构控制图形的同步渲染,同时支持分布交互式控制。这种方法在满足同等投影显示要求的情况下。将大幅度降低了系统的构建费用和维护要求。     

Projection‐based head‐tracking 3‐D displays

Abstract— This paper describes the construction and operation of four 3‐D displays in which each display produces two images for each eye and thus fits into the category of projection‐based binocular stereoscopic displays. The four 3‐D displays described are pico‐projector‐based, liquid‐ crystal—on—silicon (LCOS) conventional projector‐based, 120‐Hz digital‐light‐processor (DLP) projector‐ based, and the HELIUM3D system. In the first three displays, images are produced on a direct‐view LCD whose conventional backlight is replaced with a projection illumination source that is controlled by a multi‐user head tracker; novel steering optics direct the projector output to regions referred to as exit pupils located at the viewers' eyes. In the HELIUM3D display, the image information is supplied by a horizontally scanned, fast, light valve whose output is controlled by a spatial light modulator (SLM) to direct images to the appropriate viewers' eyes. The current statu s and the multimodal potential of the HELIUM3D display are described.     

Interactive visibility retargeting in VR using conformal visualization

In Virtual Reality, immersive systems such as the CAVE provide an important tool for the collaborative exploration of large 3D data. Unlike head-mounted displays, these systems are often only partially immersive due to space, access, or cost constraints. The resulting loss of visual information becomes a major obstacle for critical tasks that need to utilize the users' entire field of vision. We have developed a conformal visualization technique that establishes a conformal mapping between the full 360° field of view and the display geometry of a given visualization system. The mapping is provably angle-preserving and has the desirable property of preserving shapes locally, which is important for identifying shape-based features in the visual data. We apply the conformal visualization to both forward and backward rendering pipelines in a variety of retargeting scenarios, including CAVEs and angled arrangements of flat panel displays. In contrast to image-based retargeting approaches, our technique constructs accurate stereoscopic images that are free of resampling artifacts. Our user study shows that on the visual polyp detection task in Immersive Virtual Colonoscopy, conformal visualization leads to improved sensitivity at comparable examination times against the traditional rendering approach. We also develop a novel user interface based on the interactive recreation of the conformal mapping and the real-time regeneration of the view direction correspondence.     

Integrating VR and CAD

Suppliers of advanced computer-aided design and engineering applications (CAD/CAE) intend to make interactive, immersive visualization an integral part of their product offerings. Adding virtual reality (VR) capability helps reduce users' design-cycle time and cut costs. Advances in computing power and 3D visualization functionality enable this logical evolution of technology. While many VR companies have established procedures for importing data from CAD applications into VR applications, Dassault Systemes takes a different approach to integrating CAD with VR. Through collaboration with Fakespace, the company is developing native support for immersive visualization display devices within its Catia Version 5 applications. The paper discusses the functionality required to enable immersive visualization and hands-on interactivity within CAD applications. It also describes the joint development effort to bring this functionality into Catia     

3‐D crosstalk and luminance uniformity from angular luminance profiles of multiview autostereoscopic 3‐D displays

Abstract— Autostereoscopic 3‐D display technologies enable a more immersive media experience by adding real depth to the visual content. However, the method used for the creation of a sensation of depth or stereo illusion contains several display design and content‐related issues that need to be carefully considered to maintain sufficient image quality. Conventionally, methods used for 3‐D image‐quality evaluations have been based on subjective testing. Optical measurements, in addition to subjective testing, can be used as an efficient tool for 3‐D display characterization. Objective characterization methods for autostereoscopic displays have been developed. How parameters affecting stereo image quality can be defined and measured, and how their effect on the stereo image quality can be evaluated have been investigated. Developed characterization methods are based on empirically gathered data. In this paper, previously presented methodology for two‐view displays is extended to cover autostereoscopic multiview displays. A distinction between displays where the change in content occurs in clear steps when the user moves in front of the display, and displays where the apparent movement of the objects is more continuous as a function of the head movement is made. Definitions for 3‐D luminance and luminance uniformity, which are equally important, as well as 3‐D crosstalk, which is the dominant factor in the evaluations of the perceived 3‐D image quality, is focused upon.     

Laser‐based multi‐user 3‐D display

Abstract— The development of a multi‐user stereoscopic display that does not require the use of special glasses (autostereoscopic), and that enables a large degree of freedom of viewer movement and requires only the minimum amount of information (a stereo pair) for the displays described. The optics comprise an RGB holographic laser projector that is controlled by the output of a multi‐target head‐position head tracker, an optical assembly that converts the projector output into steerable exit pupils, and a screen assembly comprising a single liquid‐crystal display (LCD) and image multiplexing screen. A stereo image pair is produced on the LCD by simultaneously displaying left and right images on alternate rows of pixels. Novel steering optics that replace the conventional backlight are used to direct viewing regions, referred to as exit pupils, to the appropriate viewers' eyes. The results obtained from the first version of the display, where the illumination source consists of several thousand white LEDs, are given and the current status of the latest prototype being constructed on the basis of these results is described. The work indicates that a laser‐based head‐tracking display can provide the basis for the next generation of 3‐D display.     

注意力机制的曲面沉浸式投影系统补偿

目的 沉浸式投影系统已广泛运用于虚拟现实系统之中,然而沉浸式投影系统中的互反射现象严重影响着虚拟现实系统的落地使用。沉浸式投影系统的互反射是指由于投影机光线和屏幕反射光线相互叠加造成的亮度冗余现象,严重影响了投影系统的成像质量和人眼的视觉感受。为此,本文提出一种新的基于互反射通道(inter-reflection channel,IRC)先验和注意力机制的神经网络。方法 IRC先验基于这样一个事实,即大多数受到互反射影响的投影图像都包含一些亮度较高的区域。高亮度区域往往受互反射影响更为严重,而低亮度区域受互反射影响程度较低。根据这一规律,采用IRC先验作为注意力图的监督样本,获取补偿图像的亮度区域信息。同时,为了对投影图像不同区域按影响程度进行差异化补偿,提出一种新的由两个相同子网络构成的补偿网络结构Pair-Net。结果 实验对比了4种现有方法,Pair-Net在ROI(region of interesting)指标分析上取得了明显优势,在人眼感受上有显著的效果提升。结论 本文提出的基于注意力机制的网络模型能够针对不同区域进行差异化补偿,很大程度上消除了互反射影响,提升了沉浸式投影系统的成像质量。