An enhanced depth map based rendering method with directional depth filter and image inpainting

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D video conversion process. In this paper, a novel method is proposed to partially solve two puzzles of DIBR, i.e. visual image generation and hole filling. The method combines two different approaches for synthesizing new views from an existing view and a corresponding depth map. Disoccluded parts of the synthesized image are first classified as either smooth or highly structured. At structured regions, inpainting is used to preserve the background structure. In other regions, an improved directional depth smoothing is used to avoid disocclusion. Thus, more details and straight line structures in the generated virtual image are preserved. The key contributions include an enhanced adaptive directional filter and a directional hole inpainting algorithm. Experiments show that the disocclusion is removed and the geometric distortion is reduced efficiently. The proposed method can generate more visually satisfactory results.     

快速3维坐标变换的绘制算法

目的 基于深度图的绘制(DIBR)是一种新型的虚拟视点生成技术,在诸多方面得到了广泛的应用。然而,该技术还不能满足实时性的绘制需求。为了在保证绘制质量不下降的前提下,尽可能地提高绘制速度,提出了一种高效的3D-Warping(3维坐标变换)算法。方法 主要在以下3个方面进行了改进:1)引入了深度—视差映射表技术,避免了重复地进行视差求取操作。2)对深度平坦的像素块进行基于块的3D-Warping,减少了映射的次数。对深度非平坦像素块中的像素点采取传统的基于像素点的3D-Warping,保证了映射的准确性。3)针对两种不同的3D-Warping方式,分别提出了相应的插值算法。在水平方向上,改进的像素插值算法对紧邻插值和Splatting(散射)插值算法进行了折中,只在映射像素点与待插值像素点很近的情况下才进行紧邻插值,否则进行Splatting插值;在深度方向上,它对Z-Buffer(深度缓存)技术进行了改进,舍弃了与前景物体太远的映射像素点,而对其他映射像素点按深度值进行加权操作。结果 实验结果表明,与标准绘制方案的整像素精度相比,绘制时间平均节省了72.05%;与标准绘制方案的半像素精度相比,PSNR平均提高了0.355dB,SSIM平均提高了0.00115。结论 改进算法非常适用于水平设置相机系统的DIBR技术中的整像素精度绘制,对包含大量深度平坦区域的视频序列效果明显,不但能够提高绘制的速度,而且可以有效地改善绘制的客观质量。     

Elemental image array generation method by using optimized depth image‐based rendering algorithm for integral imaging display

We propose an elemental image array (EIA) generation method by using an optimized depth image‐based rendering (DIBR) algorithm. In this method, the EIA is synthesized by the reference and virtual viewpoint elemental images, and the virtual viewpoint elemental images at the given locations are generated by DIBR algorithm. We optimize the existing DIBR algorithm by adaptively repairing the warped depth images in the processing part and extend the generation dimension of the virtual viewpoint elemental images from one dimension to two dimensions. The optimized DIBR algorithm can effectively solve the problem: the low quality of virtual viewpoint elemental images caused by discontinuous depth values and disocclusion regions. We also implement the generations of virtual viewpoint elemental images and EIA in graph processing unit to reduce the time cost. Experimental results show that the proposed method can not only improve the quality of the virtual viewpoint images but also accelerate the generations of the virtual viewpoint elemental images and EIA.     

Direct light field rendering without 2D image generation

Rapid developments in 3D display technologies have enabled consumers to enjoy 3D environments in an increasingly immersive manner through various display systems such as stereoscopic, multiview, and light field displays. However, there is a corresponding increase in the complexity of the conventional multiview rendering process in the attempt to achieve a sufficient level of reality, which may hinder the further commercial viability of 3D display products based on such a conventional approach. This paper proposes a novel method, the so‐called direct light field rendering, which can compose the display 3D panel image without reconstructing all the multiview images beforehand. Interpreting the 3D display as sampling in the light field domain, we attempt to directly compute only the necessary samples, not the entire light fields or multiview images. Our proposed algorithm involves the solving of linear systems of two variables, thereby requiring remarkably low computational complexities. Experimental results show that the computation time and memory usage remain as little as 12% and 1% of those required by the conventional one.     

基于网格形变的立体图像内容重组

目的 近年来,随着数字摄影技术的飞速发展,图像增强技术越来越受到重视。图像构图作为图像增强中影响美学的重要因素,一直都是研究的热点。为此,从立体图像布局调整出发,提出一种基于Delaunay网格形变的立体图像内容重组方法。方法 首先将待重组的一对立体图像记为源图像,将用于重组规则确定的一幅图像记为参考图像;然后对源图像需要调整的目标、特征线和其他区域进行取点操作,建立Delaunay网格。将源图像的左图与参考图像进行模板匹配操作,得到源图像与参考图像在结构布局上的对应关系;最后利用网格形变的特性,移动和缩放目标对象,并对立体图像的深度进行自适应调整。结果 针对目标对象的移动、缩放和特征线调整几方面进行优化。当只涉及目标对象的移动或特征线调整时,立体图像视差保持不变;当目标对象缩放时,立体图像中目标对象的视差按照缩放比例变化而背景视差保持不变。实验结果表明,重组后的立体图像构图与参考图像一致且深度能自适应调整。与最新方法比较,本文方法在目标对象分割精度和图像语义保持方面具有优势。结论 根据网格形变相关理论,构建图像质量、布局匹配和视差适应3种能量项,实现了立体图像的内容重组。与现有需要提取和粘贴目标对象的重组方法不同,本文方法对目标对象的分割精度要求不高,不需要图像修复和混合技术,重组后的立体图像没有伪影和语义错误出现。用户可以通过参考图像来引导立体图像的布局调整,达到期望的图像增强效果。     

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70–0.95 (P < 0.05). A t-test was used for the statistical analysis.     

建立在图像识别基础上的体绘制加速方法

提出了一种新算法——IRVR（Image Recognition Volume Rendering）,该算法能大幅降低冗余数据,从而提升体绘制速度。IRVR算法首先利用交叉熵阈值分割法从三维数据集中将物体像素和背景像素识别出来,然后将迭代光线追踪方法和物体检测采样策略结合起来对原始三维数据集进行采样。接着运用快速迭代法对分类数据集进行采样,从而定位视线与原始数据集的交点。IRVR算法还应用了准确正规采样方法（例如,三线性插值、样条插值等）在体绘制过程中对原始数据集进行插值。经过实验得出的结论证明IRVR算法既能提高体绘制的速度,又可以保证体绘制图像的质量。     

基于OpenGL的快速图像渲染方法*

针对在动态图形绘制基础上进行图像渲染的问题,基于Mac OS X操作系统的核心制图与渲染技术,提出了重组OpenGL渲染流程实现加速图形绘制与图像渲染结合的管道化方法,目的是实现GPU完全承担绘图和渲染加速。整个流程无须CPU参与,在提高图形子系统性能的同时优化了应用程序响应能力。     

Depth image‐based rendering for multiview generation

Abstract— Techniques for 3‐D display have evolved from stereoscopic 3‐D systems to multiview 3‐D systems, which provide images corresponding to different viewpoints. Currently, new technology is required for application in multiview display systems that use input‐source formats such as 2‐D images to generate virtual‐view images of multiple viewpoints. Due to the changes in viewpoints, occlusion regions of the original image become disoccluded, resulting in problems related to the restoration of output image information that is not contained in the input image. In this paper, a method for generating multiview images through a two‐step process is proposed: (1) depth‐map refinement and (2) disoccluded‐area estimation and restoration. The first step, depth‐map processing, removes depth‐map noise, compensates for mismatches between RGB and depth, and preserves the boundaries and object shapes. The second step, disoccluded‐area estimation and restoration, predicts the disoccluded area by using disparity and restores information about the area by using information about neighboring frames that are most similar to the occlusion area. Finally, multiview rendering generates virtual‐view images by using a directional rendering algorithm with boundary blending.     

Image-based rendering of range data with estimated depth uncertainty

Image-based rendering (IBR) involves constructing an image from a new viewpoint, using several input images from different viewpoints. Our approach is to acquire or estimate the depth for each pixel of each input image. We then reconstruct the new view from the resulting collection of 3D points. When rendering images from photographs, acquiring and registering data is far from perfect. Accuracy can fluctuate, depending on the choice of geometry reconstruction technique. Our image-rendering approach involves three steps: depth extraction, uncertainty estimation, and rendering. That is, we first compute a depth map for every input image. Then we calculate the uncertainty information using the estimated depth maps as starting points. Finally, we perform the actual rendering, which renders the uncertainty estimated in the previous step as ellipsoidal Gaussian splats.     

基于强度图和深度图的多模态人脸识别

提出将全局特征表征方法2DFLD、2DPCA与局部特征表征方法LBP相结合,应用到人脸二维强度图和三维深度图进行识别;对不同分类方法的识别得分再进行归一化加权融合。对比实验结果表明,LBP对2DFLD和2DPCA的识别结果有改善作用,二维强度图和三维深度图的得分归一化加权融合对整个识别率也有一定的改善,在CASIA3D人脸数据库上的识别率最高可达94.68%。     

基于面绘制的医学序列图像三维重建

基于医学CT图像序列,利用改进的面绘制方法,分别实现了皮肤和骨骼的三维表面重建。在皮肤重建中,主要采用阈值分割法和迭代法相结合而得到光滑逼真的效果。在骨骼重建过程中,采用二值法构建骨骼三维表面,并对得到的结果进行去噪处理,进而得到清晰的骨骼轮廓。为了验证重建方法的通用性,实现了复杂结构器官--心脏的三维表面重建。实验结果表明,方法易于实现,获得了较理想的三维重建结果。重建的三维模型可进行旋转、放大和缩小等操作,这不仅有利于实现人机交互的功能,同时也有利于对模型进行设置和观察。     

Image-based streamline generation and rendering

Seeding streamlines in 3D flow fields without considering their projections in screen space can produce visually cluttered rendering results. Streamlines will overlap or intersect with each other in the output image, which makes it difficult for the user to perceive the underlying flow structure. This paper presents a method to control the seeding and generation of streamlines in image space to avoid visual cluttering and allow a more flexible exploration of flow fields. In our algorithm, 2D images with depth maps generated by a variety of visualization techniques can be used as input from which seeds are placed and streamlines are generated. The density and rendering styles of streamlines can be flexibly controlled based on various criteria to improve visual clarity. With our image space approach, it is straightforward to implement the level of detail rendering, depth peeling, and stylized rendering of streamlines to allow for more effective visualization of 3D flow fields.     

基于带squeeze-and-excitation模块的ResNeXt的单目图像深度估计方法

针对目前单目图像深度估计任务缺乏对特征通道之间的全局信息关系表示的问题,提出了一种基于SE-ResNeXt的单目图像深度估计方法。首先,通过建模特征通道间的动态且非线性的关系来提高网络的全局信息表示能力;然后,采用特征重标定策略来自适应地重新校准特征通道的响应,从而进一步提升特征利用率;最后,通过ResNeXt结构在不增加模型复杂度的基础上进一步提升方法的性能。实验结果表明,相比与没有采用ResNeXt结构的算法,该方法获得了更低的误差值,其均方根误差（RMSE）降低了10%,绝对相对误差（AbsRel）降低了27%。     

基于深度数据的人体动作识别方法

本文提出了一个基于流形学习的动作识别框架,用来识别深度图像序列中的人体行为。本文从Kinect设备获得的深度信息中评估出人体的关节点信息,并用相对关节点位置差作为人体特征表达。在训练阶段,本文利用Lapacian eigenmaps(LE)流形学习对高维空间下的训练集进行降维,得到低维隐空间下的运动模型。在识别阶段,本文用最近邻差值方法将测试序列映射到低维流形空间中去,然后进行匹配计算。在匹配过程中,通过使用改进的Hausdorff距离对低维空间下测试序列和训练运动集的吻合度和相似度进行度量。本文用Kinect设备捕获的数据进行了实验,取得了良好的效果;同时本文也在MSR Action3D数据库上进行了测试,结果表明在训练样本较多情况下,本文识别效果优于以往方法。实验结果表明本文所提的方法适用于基于深度图像序列的人体动作识别。     

基于Kinect深度图像的三维重建

随着机器视觉理论的发展和硬件技术的进步,三维重建在生产、生活中的应用越来越广泛,基于Kinect传感器的三维重建得到广泛的应用。针对于现有的Kinect传感器获得的深度图像深度信息丢失的问题,提出了一种新的基于均值滤波的方法对深度图像进行去噪,并对深度图像进行预处理,获取三维点云,用迭代最近点(Iterative Closest Point,ICP)算法完成点云的精确配准,从而得到配准后物体表面三维点云,并完成物体的三维重建。     

Stereoscopic depth perception survives significant interocular luminance differences

Abstract— The effects of interocular luminance differences on stereoscopic depth perception has been investigated. The stimuli were stereoscopic square‐shaped targets created from disparity embedded in a dynamic random‐dot stereogram, which eliminated other cues to depth perception. The results revealed that stereoscopic depth perception survived significant interocular differences in luminance levels, even up to differences of 60%, provided that display luminance was approximately 0.63 cd/m² or higher. In terms of design criteria, developers of stereo displays can use a fairly large range of interocular luminance levels and still induce good stereo depth perception.     

基于 CNN 的彩色图像引导的深度图像超分辨率重建

深度图像表达了三维场景内物体之间的相对距离信息,根据深度图像表达的信息, 人们能够准确的获得物体在空间中的位置以及不同物体之间的相对距离,使得深度图像在立体 视觉等领域有着广泛的应用。然而受 RGB-D 传感器硬件条件的限制,获取的深度图像分辨率 低,无法满足一些具有高精度要求的实际应用的需求。近年来深度学习特别是卷积神经网络 (CNN)在图像处理方面获得了非常大的成功。为此提出了一种基于 CNN 的彩色图像引导的深度 图像超分辨率重建。首先,利用 CNN 学习彩色图像的边缘特征信息与深度图像的深度特征信 息,获得边缘纹理清晰的高分辨率深度图像;再通过不同大小尺寸滤波核的卷积层,进一步优 化深度图像的边缘纹理细节,获得更高质量的高分辨率深度图像。实验结果表明,相较于其他 方法,该方法 RMSE 值更低,重建的图像也能更好的恢复图像边缘纹理细节。     

基于深度图像绘制技术的正反向映射技术的改进

针对基于双向深度图像绘制技术（Double-sided Depth-Image Based Rendering,Double-sided DIBR）中产生的空洞、重采样、重叠问题,为提高虚拟图像的合成质量,提出一种改进的正反向映射技术。该技术主要有四点贡献。（1）提出一种深度差值估计法。（2）在3D-warping过程中使用改进的基于Z-buffer的OPFD算法,有效解决重采样和重叠问题。（3）对深度虚拟图像运用改进的基于背景空洞填补算法消除空洞。（4）改进反向映射过程,通过判断投影后的图像和辅助彩色参考图像被遮挡信息背景的一致性,选择不同的空洞填补算法填补彩色虚拟图像中的空洞,从而达到更好的填补效果。实验结果表明,改进技术在降低算法复杂度的同时,主观图像质量与客观峰值信噪比（Peak Signal to Noise Ratio,PSNR）以及结构相似（Structural SIMilarity,SSIM）都有所提高。