Attention-Aware Disparity Control in interactive environments

Our paper introduces a novel approach for controlling stereo camera parameters in interactive 3D environments in a way that specifically addresses the interplay of binocular depth perception and saliency of scene contents. Our proposed Dynamic Attention-Aware Disparity Control (DADC) method produces depth-rich stereo rendering that improves viewer comfort through joint optimization of stereo parameters. While constructing the optimization model, we consider the importance of scene elements, as well as their distance to the camera and the locus of attention on the display. Our method also optimizes the depth effect of a given scene by considering the individual user’s stereoscopic disparity range and comfortable viewing experience by controlling accommodation/convergence conflict. We validate our method in a formal user study that also reveals the advantages, such as superior quality and practical relevance, of considering our method.     

基于人眼立体视觉特性的多视视频码率控制算法

提出了一种基于人眼立体视觉特性的多视视频码率控制算法.引入了立体视觉质量的客观评价标准,并基于此推导出一种帧级目标码率分配方法,同时根据相邻宏块量化参数来调整当前宏块量化参数,以消除立体视频的块效应.仿真结果显示,采用本文算法的多视视频编码器,在取得更稳定的比特率的同时,有效地提高了立体视觉质量.     

Effects of 3D perspective on head gaze estimation with a multiview autostereoscopic display

Head gaze, or the orientation of the head, is a very important attentional cue in face to face conversation. Some subtleties of the gaze can be lost in common teleconferencing systems, because a single perspective warps spatial characteristics. A recent random hole display is a potentially interesting display for group conversation, as it allows multiple stereo viewers in arbitrary locations, without the restriction of conventional autostereoscopic displays on viewing positions. We represented a remote person as an avatar on a random hole display. We evaluated this system by measuring the ability of multiple observers with different horizontal and vertical viewing angles to accurately and simultaneously judge which targets the avatar is gazing at. We compared three perspective conditions: a conventional 2D view, a monoscopic perspective-correct view, and a stereoscopic perspective-correct views. In the latter two conditions, the random hole display shows three and six views simultaneously. Although the random hole display does not provide high quality view, because it has to distribute display pixels among multiple viewers, the different views are easily distinguished. Results suggest the combined presence of perspective-correct and stereoscopic cues significantly improved the effectiveness with which observers were able to assess the avatar׳s head gaze direction. This motivates the need for stereo in future multiview displays.     

Optimizing stereo‐to‐multiview conversion for autostereoscopic displays

We present a novel stereo‐to‐multiview video conversion method for glasses‐free multiview displays. Different from previous stereo‐to‐multiview approaches, our mapping algorithm utilizes the limited depth range of autostereoscopic displays optimally and strives to preserve the scene's artistic composition and perceived depth even under strong depth compression. We first present an investigation of how perceived image quality relates to spatial frequency and disparity. The outcome of this study is utilized in a two‐step mapping algorithm, where we (i) compress the scene depth using a non‐linear global function to the depth range of an autostereoscopic display and (ii) enhance the depth gradients of salient objects to restore the perceived depth and salient scene structure. Finally, an adapted image domain warping algorithm is proposed to generate the multiview output, which enables overall disparity range extension.     

一种可处理数据缺失的视角无关手语识别方法

基于虚拟立体视假设,借鉴RANSAC技术的思想,文中针对数据缺失(帧对之间匹配特征可能较少)情况下的视角无关手语识别问题,提出一种Sample-Consensus方法.其基本出发点是,同一手语不同视角下的两个样本序列之间所有的对应帧对,可以解释为由某一虚拟立体视觉系统同步捕获,因而满足同一个基础矩阵,而且此基础矩阵能够基于部分对应帧对包含的点对应关系进行估计.实验表明,提出的Sample-Consensus方法能够有效地应用于数据缺失情况下的视角无关手语识别.另外,这种方法也可以扩展到相近的领域,如视角无关的动作识别和刚体运动分析等.     

人眼立体视觉空间与虚拟三维空间的几何映射关系理论分析

虚拟三维空间是现实世界的数字化三维空间,而人眼立体视觉空间则是人眼视觉系统对于现实世界或虚拟世界所形成的三维立体构象。传统上人眼直接观察现实世界,确立了人眼立体视觉空间与现实世界之间的几何对应关系。而人眼立体视觉空间与虚拟三维空间是否也存在对应的几何关系？以人眼立体视觉和虚拟三维场景为研究对象,根据双目视差原理,论述了人眼立体视觉的几何模型及视觉三维模型的表示形式。通过分析虚拟空间三维点元、屏幕视差及网膜视差等三者之间的内在几何关系,利用矩阵代数建立了虚拟空间与视觉空间之间的几何映射关系。这一映射关系表明视觉三维模型与虚拟三维模型之间存在一一对应关系,也反映了人眼视觉系统在虚拟空间中的可测量性质。本文创新性之处在于得到了视觉三维模型的完整表示,突破了传统上立体显示的定性感知,提供了定量分析的基础。这一工作对于在虚拟空间中的立体体验、虚拟交互以及立体测量等实践活动具有一定理论参考价值。     

Development of dense models of the Earth’s surface based on the semiglobal matching method for the case of multiple overlapping images

A method is developed for constructing a three-dimensional digital surface model based on the use of aerial images with multiple overlapping. The specific feature of this study is the use of the multiview matching method instead of stereo matching. The method is based on adapting the energy aggregation algorithm, which was proposed in the semiglobal matching (SGM) method, to the object space, as well as using the one-to-many scheme of cost calculation. The reconstructed scene is represented as a voxel grid. A high-performance implementation of the digital surface model construction at all stages is proposed based on the massive parallelization of computations on a graphics processing unit.     

Stereo‐Consistent Contours in Object Space

Notebook scribbles, art or technical illustrations—line drawings are a simplistic method to visually communicate information. Automated line drawings often originate from virtual 3D models, but one cannot trivially experience their three‐dimensionality. This paper introduces a novel concept to produce stereo‐consistent line drawings of virtual 3D objects. Some contour lines do not only depend on an objects geometry, but also on the position of the observer. To accomplish consistency between multiple view positions, our approach exploits geometrical characteristics of 3D surfaces in object space. Established techniques for stereo‐consistent line drawings operate on rendered pixel images. In contrast, our pipeline operates in object space using vector geometry, which yields many advantages: The position of the final viewpoint(s) is flexible within a certain window even after the contour generation, e.g. a stereoscopic image pair is only one possible application. Such windows can be concatenated to simulate contours observed from an arbitrary camera path. Various types of popular contour generators can be handled equivalently, occlusions are natively supported and stylization based on geometry characteristics is also easily possible.     

一种基于棱柱镜的立体图像二维抗混叠数字滤波方法

基于棱柱镜LCD自由立体显示器及其立体图像的特征,针对频域混叠失真问题,提出了一种新的二维抗混叠数字滤波方法。该方法分析了倾斜棱柱镜光栅下多视点LCD子像素排列规律,并结合多维采样理论建立了子采样模型,设计出相应的二维数字滤波器,从而对各视点图像进行二维频域抗混叠滤波以消除失真。实验证明,该方法复杂度低、滤波效果好,对同类自由立体显示器具有适用性。     

Depth image‐based rendering for multiview generation

Abstract— Techniques for 3‐D display have evolved from stereoscopic 3‐D systems to multiview 3‐D systems, which provide images corresponding to different viewpoints. Currently, new technology is required for application in multiview display systems that use input‐source formats such as 2‐D images to generate virtual‐view images of multiple viewpoints. Due to the changes in viewpoints, occlusion regions of the original image become disoccluded, resulting in problems related to the restoration of output image information that is not contained in the input image. In this paper, a method for generating multiview images through a two‐step process is proposed: (1) depth‐map refinement and (2) disoccluded‐area estimation and restoration. The first step, depth‐map processing, removes depth‐map noise, compensates for mismatches between RGB and depth, and preserves the boundaries and object shapes. The second step, disoccluded‐area estimation and restoration, predicts the disoccluded area by using disparity and restores information about the area by using information about neighboring frames that are most similar to the occlusion area. Finally, multiview rendering generates virtual‐view images by using a directional rendering algorithm with boundary blending.     

Stereo panorama acquisition and automatic image disparity adjustment for stereoscopic visualization

Image-based visualization is popular for various virtual tour applications, due to high-quality photorealism or simplicity for rendering. Stereo panorama representations of the virtual world are already a common part of this, either in small (computer screen) format or on large-scale stereo displays or screens. This paper discusses methods for determining optimum parameters, both for high-accuracy stereo panoramic image recording and displaying, with a special focus on automatic image disparity enhancement while displaying (e.g., including zooming) a stereo panorama. Experiments show that the discussed parameters are indeed critical for ensuring high-quality stereo viewing. Derived formulas in this study are applicable to various kinds of technologies for stereo panorama imaging or stereoscopic displaying.     

On 3-D scene flow and structure recovery from multiview image sequences

Two novel systems computing dense three-dimensional (3-D) scene flow and structure from multiview image sequences are described in this paper. We do not assume rigidity of the scene motion, thus allowing for nonrigid motion in the scene. The first system, integrated model-based system (IMS), assumes that each small local image region is undergoing 3-D affine motion. Non-linear motion model fitting based on both optical flow constraints and stereo constraints is then carried out on each local region in order to simultaneously estimate 3-D motion correspondences and structure. The second system is based on extended gradient-based system (EGS), a natural extension of two-dimensional (2-D) optical flow computation. In this method, a new hierarchical rule-based stereo matching algorithm is first developed to estimate the initial disparity map. Different available constraints under a multiview camera setup are further investigated and utilized in the proposed motion estimation. We use image segmentation information to adopt and maintain the motion and depth discontinuities. Within the framework for EGS, we present two different formulations for 3-D scene flow and structure computation. One formulation assumes that initial disparity map is accurate, while the other does not. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3-D motion and structure recovery schemes. Empirical comparison between IMS and EGS is also reported.     

Interactive stereoscopic displays

Interactive stereo displays allow for the existence of a natural interaction between the user and the stereo images depicted on the display. In the type of display discussed here, this interaction takes the form of tracking the user's head and hand/arm position. Sensing the user's head position allows for the creation of motion parallax information, an immersive depth cue that can be added to the binocular parallax already present in the display. Sensing the user's hand or arm position allows the user to manipulate the spatial attributes of virtual objects and scenes presented on the display, which can enhance spatial reasoning. Moreover, allowing the user to manipulate virtual objects may permit the creation of a sense of spatial relations among elements in the display via proprioception, which may augment the two parallax cues. The congruence among binocular parallax, motion parallax, and proprioception should increase the sense of depth in the display and increase viewing comfort, as well as enhance the ability of our intuitive reasoning system to make reasoned sense out of the perceptual information. These advantages should make interactive stereo displays, which may be classified as a form of cognitive enhancement display, the display of choice in the future.     

Effects of interface and spatial ability on manipulation of virtual models in a STEM domain

Virtual models are increasingly employed in STEM education to foster learning about spatial phenomena. However, the roles of the computer interface and students’ cognitive abilities in moderating learning and performance with virtual models are not yet well understood. In two experiments students solved spatial organic chemistry problems using a virtual model system. Two aspects of the virtual model interface were manipulated: display dimensionality (stereoscopic vs. monoscopic displays) and the location of the hand-held device used to manipulate the virtual molecules (co-located with the visual display vs. displaced). The experimental task required participants to interpret the spatial structure of organic molecules and to manipulate the models to align them with orientations and configurations depicted by diagrams in Experiment 1 and three-dimensional models in Experiment 2. Co-locating the interaction device with the virtual image led to better performance in both experiments and stereoscopic viewing led to better performance in Experiment 2. The effect of co-location on performance was moderated by spatial ability in Experiment 1, and the effect of providing stereo viewing was moderated by spatial ability in Experiment 2. The results are in line with the ability-as-compensator hypothesis: participants with lower ability uniquely benefited from the treatment, while those with higher ability were not affected by stereo or co-location. The findings suggest that increased fidelity in a virtual model system may be one way of alleviating difficulties of low-spatial participants in learning spatially demanding content in STEM domains.     

Dual-view holographic AR display based on Bragg mismatched reconstruction of the holographic optical element

A method for dual-view holographic display based on Bragg mismatched reconstruction of holographic optical element (HOE) is proposed. Under the Bragg mismatched condition, the reconstructed images are guided into two separated viewing zones to realize dual-view holographic display. Meanwhile, the viewing angle of each perspective is increased to 11.2°, which is almost 2.5 times as large as the traditional holographic display system. The design process of HOE is simple only by interference of plane reference wave and converging spherical signal wave, which has high practicability. Furthermore, the HOE can mix the virtual 3D image with real-world scenes, which could implement augmented reality (AR) display. Experiments validate that the proposed system can achieve dual-view holographic AR three-dimensional (3D) display with accommodation effect.     

Comprehensive Facial Performance Capture

We present a system for recording a live dynamic facial performance, capturing highly detailed geometry and spatially varying diffuse and specular reflectance information for each frame of the performance. The result is a reproduction of the performance that can be rendered from novel viewpoints and novel lighting conditions, achieving photorealistic integration into any virtual environment. Dynamic performances are captured directly, without the need for any template geometry or static geometry scans, and processing is completely automatic, requiring no human input or guidance. Our key contributions are a heuristic for estimating facial reflectance information from gradient illumination photographs, and a geometry optimization framework that maximizes a principled likelihood function combining multi‐view stereo correspondence and photometric stereo, using multi‐resolution belief propagation. The output of our system is a sequence of geometries and reflectance maps, suitable for rendering in off‐the‐shelf software. We show results from our system rendered under novel viewpoints and lighting conditions, and validate our results by demonstrating a close match to ground truth photographs.     

Joint Estimation of Shape and Reflectance using Multiple Images with Known Illumination Conditions

We propose a generative model based method for recovering both the shape and the reflectance of the surface(s) of a scene from multiple images, assuming that illumination conditions and cameras calibration are known in advance. Based on a variational framework and via gradient descents, the algorithm minimizes simultaneously and consistently a global cost functional with respect to both shape and reflectance. The motivations for our approach are threefold. (1) Contrary to previous works which mainly consider specific individual scenarios, our method applies indiscriminately to a number of classical scenarios; in particular it works for classical stereovision, multiview photometric stereo and multiview shape from shading. It works with changing as well as static illumination. (2) Our approach naturally combines stereo, silhouette and shading cues in a single framework. (3) Moreover, unlike most previous methods dealing with only Lambertian surfaces, the proposed method considers general dichromatic surfaces. We verify the method using various synthetic and real data sets.     

Advances in the Dynallax solid-state dynamic parallax barrier autostereoscopic visualization display system

A solid-state dynamic parallax barrier autostereoscopic display mitigates some of the restrictions present in static barrier systems, such as fixed view-distance range, slow response to head movements, and fixed stereo operating mode. By dynamically varying barrier parameters in real time, viewers may move closer to the display and move faster laterally than with a static barrier system, and the display can switch between 3D and 2D modes by disabling the barrier on a per-pixel basis. Moreover, Dynallax can output four independent eye channels when two viewers are present, and both head-tracked viewers receive an independent pair of left-eye and right-eye perspective views based on their position in 3D space. The display device is constructed by using a dual-stacked LCD monitor where a dynamic barrier is rendered on the front display and a modulated virtual environment composed of two or four channels is rendered on the rear display. Dynallax was recently demonstrated in a small-scale head-tracked prototype system. This paper summarizes the concepts presented earlier, extends the discussion of various topics, and presents recent improvements to the system.     

多视三维重构中连续能量模型的凸优化

利用融合了轮廓线及体视的序列图像信息,提出了一个面向多视三维重构的稳健能量模型;为了适配于可视性约束,提出一种针对该能量模型的连续全局优化方法;为了保证栅格连通性选择的一致性及独立性,实施了全局连续优化的超松弛离散化。实例证明,该方法的实用性好,极大地减少了算法处理的内存开销,实现了在更高分辨率上有效的多视重构。     

基于网络/主机副本法的诱骗系统的研究与实现

分析比较了现有的网络诱骗技术及其优缺点,在采用网络/主机副本方案的基础上,提出了一种基于Limux动态进程共享库注射的数据捕获方法.系统主要分为诱骗客户端和服务器两个部分,客户端主要负责数据捕获和数据发送,服务器端将获得的数据存储并分类显示,显示方式包括命令行和图形界面.系统还采用了用户态-内核态通信、内核态发包、内核模块隐藏等技术.