基于逆向映射的空洞填补方法

为了有效填补虚拟视点图像中的公共空洞,提出 了一种基于逆向映射的空洞填补方法。 首先利用深度图像绘制(DIBR)技术将左、右参考视点映射到虚拟视点位置,利用图像膨胀方 法将映射的虚拟视图中的空 洞区域进行扩大,以消除虚拟视点图像中的伪影瑕疵;然后,提取出膨胀后空洞区域的边界 ,并将其逆映 射到原始的参考图像中,根据空洞与边界的相对位置,选取原始图像中相对位置上的像素来 填充虚拟视图 中的空洞区域;最后,将空洞填补之后的左、右视点映射的虚拟视图进行融合获得最终的虚 拟视图。实验 证明,本文方法有效解决了传统空洞填补方法容易将前景像素填充到背景区域的问题,能 够获得较好的视觉观看效果和较高的客观峰值信噪比(PSNR)值。     

一种基于时间导向的DIBR空洞填补方法

在立体成像系统中,填充新视点合成时图像中产生的空洞是虚拟视点合成技术的重要环节,已有方法是对空洞区域周围的像素进行水平的推测运算来得到填洞像素,其正确率不高而且容易造成图像边缘失真。提出一种基于时间导向的DIBR空洞填补方法,首先通过结构相似帧的方法得到与关键帧拥有相同背景的前后帧图像,然后比较空洞位置对应的前后帧图像的深度值,利用背景信息对空洞区域进行填补。实验结果表明,新方法得到的填补后的图像效果可以较为真实地还原特定的场景,与原始视点的结构相似性可达到91.36%,比传统方法提高了8.38%。     

多视绘制中的空洞填充算法

绘制新视点的质量决定3D视频在显示终端的效果,为填充基于深度图像的绘制(DIBR)算法中产生的空洞,本文提出"双路纹理+双路深度"的多视绘制算法。首先,应用DIBR技术,通过左侧参考纹理图像和其对应的深度图像绘制虚拟视点图像,从经中值滤波后的虚拟图像绘制空洞掩膜图像;然后,将掩膜图像中的大空洞点坐标反变换到右侧参考纹理图像中对应的具体像素坐标,根据深度值判断得到的像素点是否属于背景区域,以此得到虚拟视点图像的空洞填充图像;最后,将空洞填充图像与左视经过DIBR得到的虚拟图像进行融合,填补大空洞,应用插值算法填充小的空洞。实验结果表明,本文方法可有效修复DIBR绘制过程中产生的空洞,得到质量较好的虚拟视点图像。     

基于卷积神经网络的虚拟视点空洞填补

为了提高虚拟视点绘制中空洞填补的质量,本文提 出一种基于卷积神经网络的虚拟视点空洞填补算 法,包括图像预处理、特征提取和空洞填补三部分。首先,对双视点虚拟视点图像进行预处 理,即利用深 度图像绘制(Depth Image Based Rendering,DIBR)技术得到空洞填补前的图像和空洞掩膜 ;然后,通过卷积 神经网络对虚拟视点图像提取多维特征,针对空洞区域,使用结构相似性损失函数优化网络 ;最后,利用 提取的特征填补虚拟视点中的空洞。实验结果表明,本文算法能较好地保持虚拟视点图像前 景和背景的锐 利边缘,主观视觉感知效果佳,同时,对不同场景测试序列的客观评价指标PS NR和SSIM都能取得满意结果,该方法能有效地填补虚拟视点中的空洞。     

基于虚拟曝光图像的立体高动态范围图像合成算法

本文提出基于多视点多曝光图像的立体高动态范围图像合成算法。首 先,考虑多视点多曝光 图像以及相机响应函数曲线的特性,提出一种虚拟曝光图像绘制算法,将不同曝光的图像绘 制到同一视点;然后, 为了使绘制曝光图像保留更多细节和结构,需要对绘制虚拟曝光图像进行空洞填补及边缘修 复,故引入了边缘差值 掩膜图,对图像边缘信息进行校正平滑处理;最后利用绘制的虚拟曝光图像合成立体高动态 范围图像。实验结果表 明,获得的绘制曝光图像与参考曝光视点图像之间的结构相似性高达0.99以上,且合成的 高动态范围图像质量高。     

高质量的虚拟视点图像的绘制方法

针对应用深度图像的绘制(DIBR)技术生成的虚拟视点图像中存在的空洞问题,提出了一种基于DIBR的高质量的虚拟视点图像绘制算法.先由多幅参考图像分别生成同一虚拟视点位置的多幅目标图像,将这些虚拟视点目标图像融合为含有少量空洞的目标图像.然后采用逆映射的方法在多幅参考图像中找到这些剩余少量空洞的对应内容并填充空洞,从而生成高质量的虚拟视点图像.实验结果表明最后生成的虚拟视点图像具有良好的视觉效果.     

基于背景提取和分区修复的DIBR空洞填补方法

针对基于深度图的视点绘制中空洞填补时产生的图像失真问题,提出一种还原空洞区域可靠纹理的空洞填补方法。通过从临近的帧画面中提取有效信息还原真实的背景图像,用于新视点中背景区域的空洞填补。并通过区分前景和背景图像区域,对不同区域的空洞采取不同的填充方法,从而获得较为可靠和真实的填补效果。从实验结果来看,该方法获得的PSNR数据优于其他算法,并且与真实视点比较的结构相似度达到99.1%,比传统方法提高10%以上。     

基于Kinect的实时深度提取与多视绘制算法

提出了一种基于Kinect的实时深度提取算法和单纹理+深度的多视绘制方法。在采集端,使用Kinect提取场景纹理和深度,并针对Kinect输出深度图的空洞提出一种快速修复算法。在显示端,针对单纹理+深度的基于深度图像的绘制(DIBR,depth image based rendering)绘制产生的大空洞,采用一种基于背景估计和前景分割的绘制方法。实验结果表明,本文方法可实时提取质量良好的深度图,并有效修复了DIBR绘制过程中产生的大空洞,得到质量较好的多路虚拟视点图像。以所提出的深度获取和绘制算法为核心,实现了一种基于深度的立体视频系统,最终的虚拟视点交织立体显示的立体效果良好,进一步验证了本文算法的有效性。本文系统可用于实景的多视点立体视频录制与播放。     

基于视点相关性的光场图像压缩算法

为了探索虚拟绘制视点之间的强相关性, 提高光场图像的压缩效率, 提出一种基于视点相关性的光场图像压缩算法。该算法基于高清视频编码屏幕内容编码扩展平台, 利用线性加权算法以及帧内块拷贝混合预测算法来提升编码块的预测精度; 并利用率失真优化过程来自适应地选择最优的编码块大小以及预测模式。结果表明, 所提算法相比于高清视频编码标准可以获得2.55dB的平均BD-峰值信噪比编码增益, 同时可以获得较好的虚拟视点绘制质量。该算法充分利用虚拟绘制视点之间的强相关性, 提高了光场图像的编码效率。     

基于几何模型的高质量虚拟视点绘制

该研究主要完成用kineet获得的“单路纹理+深度图像”来生成新的虚拟视点图像.针对深度图像的虚拟视点绘制技术(Depth Image Based Rendering,DIBR)生成的虚拟图像空洞问题,利用高斯混合模型分离前背景,背景空洞采用背景值填充,前景空洞采用改进的图像修复技术方法来填充.实验证明生成的虚拟视点图像具有较好的视觉效果.     

High efficiency depth image-based rendering with simplified inpainting-based hole filling

Hole and crack filling is the most important issue in depth-image-based rendering (DIBR) algorithms for generating virtual view images when only one view image and one depth map are available. This paper proposes a priority patch inpainting algorithm for hole filling in DIBR algorithms by generating multiple virtual views. A texture-based interpolation method is applied for crack filling. Then, an inpainting-based algorithm is applied patch by patch for hole filling. A prioritized method for selecting the critical patch is also proposed to reduce computation time. Finally, the proposed method is realized on the compute unified device architecture parallel computing platform which runs on a graphics processing unit. Simulation results show that the proposed algorithm is 51-fold faster for virtual view synthesis and achieves better virtual view quality compared to the traditional DIBR algorithm which contains depth preprocessing, warping, and hole filling.     

A robust blind watermarking algorithm for depth-image-based rendering 3D images

The protection of 3D contents from illegal distribution has attracted considerable attention and depth-image-based rendering (DIBR) is proved to be a promising technology for 3D image and video displaying. In this paper, we propose a new digital watermarking scheme for DIBR 3D images based on feature regions and ridgelet transform (RT). In this scheme, the center view and the depth map are made available at the content provider side. After selecting the reference points of the center view, we construct the feature regions for watermark embedding. Considering the sparse image representation and directional sensitivity of the RT, the watermark bits are embedded into the amplitudes of the ridgelet coefficients of the most energetic direction. The virtual left and virtual right views are generated from the watermarked center view and the associated depth map at the content consumer side. The watermarked view has good perceptual quality under both the objective and subjective image quality evaluations. The embedded watermark can be detected blindly with low bit error rate (BER) from the watermarked center view, the synthesized left and right views even when the views are distorted and distributed separately. The experimental results demonstrate that the proposed scheme exhibits good performance in terms of robustness against various image processing attacks. Meanwhile, our method can be robust to common DIBR processing, such as depth image variation, baseline distance adjustment and different rendering conditions. Furthermore, compared with other related and state-of-the-art methods, the proposed algorithm shows higher accuracy in watermark extraction.     

Depth map misalignment correction and dilation for DIBR view synthesis

The quality of the synthesized views by Depth Image Based Rendering (DIBR) highly depends on the accuracy of the depth map, especially the alignment of object boundaries of texture image. In practice, the misalignment of sharp depth map edges is the major cause of the annoying artifacts at the disoccluded regions of the synthesized views. Conventional smooth filter approach blurs the depth map to reduce the disoccluded regions. The drawbacks are the degradation of 3D perception of the reconstructed 3D videos and the destruction of the texture in background regions. Conventional edge preserving filter utilizes the color image information in order to align the depth edges with color edges. Unfortunately, the characteristics of color edges and depth edges are very different which causes annoying boundaries artifacts in the synthesized virtual views. Recent solution of reliability-based approach uses reliable warping information from other views to fill the holes. However, it is not suitable for the view synthesis in video-plus-depth based DIBR applications. In this paper, a new depth map preprocessing approach is proposed. It utilizes Watershed color segmentation method to correct the depth map misalignment and then the depth map object boundaries are extended to cover the transitional edge regions of color image. This approach can handle the sharp depth map edges lying inside or outside the object boundaries in 2D sense. The quality of the disoccluded regions of the synthesized views can be significantly improved and unknown depth values can also be estimated. Experimental results show that the proposed method achieves superior performance for view synthesis by DIBR especially for generating large baseline virtual views.     

Multi-layer and Multi-scale feature aggregation for DIBR-Synthesized image quality assessment

Depth-Image-Based Rendering (DIBR) is one of the main fundamental techniques for generating new viewpoints in 3D video applications such as multi-viewpoint video (MVV), free viewpoint video (FVV) and virtual reality (VR). Due to the imperfections of color images, depth maps or texture restoration techniques, several types of distortions occur in synthesized views. However, most of related works evaluated the quality of DIBR-synthesized views by only detecting a specific type of distortion, such as stretching, black holes, blurring, etc., which were unable to accurately evaluate the quality of DIBR-synthesized views. In this paper, a new no-reference image quality assessment method is proposed to evaluate the quality of DIBR-synthesized images by combining multi-layer and multi-scale features of images. To be specific, the distortions introduced by different stages of virtual viewpoint synthesis are first analyzed, and then multi-layer and multi-scale features are extracted to estimate the degree of texture and structure distortions. As a result, individual quality scores associated with two types of distortions (e.g., structural distortion and texture distortion) are aggregated to an overall image quality. Experimental results on two publicly available DIBR datasets show that the method has better performance than the state-of-the-art models.Index Terms: image quality assessment, DIBR-synthesized image, distortion correction, BIQA.     

Stereoscopic view synthesis based on region-wise rendering and sparse representation

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D conversion process. One of the main problems in DIBR is how to reduce holes that occur on the generated virtual view images. In this paper, we make two main contributions to deal with the problem. Firstly, a region-wise rendering framework, which divides the original image regions into three special classes and renders each with optimal adaptive process respectively, is introduced. Then, a novel sparse representation-based inpainting method, which can yield visually satisfactory results with less computational complexity for high quality 2D to 3D conversion, is proposed. Numerical experimental results demonstrate the good performance of the proposed methods.     

基于GPU加速的深度图像绘制

基于深度图像的绘制(DIBR)广泛应用于虚拟视点的合成,但是目前实现DIBR的算法复杂度都比较高,很难较实时地应用到3DTV系统中。采用单路纹理图像和其对应的深度图像进行虚拟视点的合成,在图形处理单元(GPU)上应用CUDA(Compute Unified Device Architecture)技术实现了基于深度图像的绘制。通过在NVIDIA Telsa C2050图形卡上运行,绘制分辨力1 024×768和640×480的图像速率分别达到了15 f/s(帧/秒)和24 f/s,分别能够准实时或实时地应用到3DTV系统中;同时本文的绘制方法有效地节约了传输带宽,绘制图像的主观质量良好。     

Virtual view synthesis using joint information from multi-view

Immersive media has attracted widespread attention with the development of virtual reality. Three Degree of Freedom Plus media greatly enhances the user experience by allowing users’ head motion and viewpoint switching within a certain range. Due to the limitation of panoramic video acquisition and transmission, it is impossible to obtain videos from any viewpoint directly. Virtual view synthesis is the general solution to this problem. However, existing algorithms do not adequately consider the pixel correlation between multiple views. Thus, we propose a virtual view synthesis algorithm using joint information from multi-view panoramic videos to further explore the pixel correlation. Specifically, sub-pixels from different reference views in the virtual view are obtained by performing multi-view three-dimensional image warping. Dedicated area division and interpolation methods are then designed to improve the synthesized quality. Experimental results show that the proposed algorithm outperforms the state-of-the-art virtual view synthesis algorithms in performance and efficiency.     

基于图像分割的深度视频校正算法

在基于深度图的虚拟视点绘制过程中,由于通过深 度估计软件获取的深度视频存在大量的失真,从而导致绘制的虚拟视点中存在纹理失真和缺 失现象。本文围绕深度视频失真类型,提出一种基于分割的深度 视频校正算法。利用彩色深度一致性信息分区域校正深度失真,以解决由于深度块失真造成 的虚拟视点纹理 缺失问题。首先,提取彩色视频运动和边缘区域,得到彩色视频边缘和运动区域掩模图;其 次,在边缘和运 动信息的辅助下,对彩色图像进行Mean Shift聚类,并将不同类别区域赋以不同的标签;最 后,分别统计不 同类别连通区域对应的深度直方图,利用其峰值校正深度视频中深度彩色非一致区域。实验 结果表明,本文提 出的基于分割块的深度视频校正算法优于部分基于像素的滤波算法,可以有效地校正深度视 频块失真,解决 虚拟视点边缘失真和纹理缺失问题,同时虚拟视点质量平均提高了0.20dB。