Hierarchical visual comfort assessment for stereoscopic image retargeting

Stereoscopic Image Retargeting (SIR) has made it possible for the popularity of 3D application. Meanwhile, the adjustments brought to images may affect the visual comfort when enjoying 3D service. While for SIR, previous Visual Comfort Assessment (VCA) methods often cannot perform well, because they only analyze the influence of disparity on discomfort and do not take into account the effects from the unique and complex distortions of SIR. In this paper, we propose a Hierarchical Visual Comfort Assessment (Hi-VCA) scheme for SIR, considering hybrid distortions including structure, information, semantic distortions usually occurring in retargeting, and binocular incongruity existing in stereoscopic multimedia. Specifically, we first propose valid Local-SSIM and Dual Natural Scene Statistics (D-NSS) features to measure structural distortion and information loss. Considering disparity adjustments may brought by SIR, we design the binocular incongruity measurement by analyzing various binocular anomaly perception mechanisms of HVS. Finally, CNN-based feature is utilized to ensure the correct delivery of semantic information. Each measurement is complementary in describing visual comfort degradation and they are further aggregated. Extensive experiment results on published SIR database SIRD and two ordinary databases IEEE-SA and NBU 3D-VCA, demonstrate Hi-VCA has superior performance by better handling hybrid distortions compared to state-of-the-art schemes.     

LED电视在3D模式下的视觉健康舒适度测试研究

通过客观量测人眼观视时的各项视觉生理指标,并将各项指标嵌入人眼健康舒适度评价模型进行分析运算的方式,经由180人次的人因实验,从视觉健康舒适度等方面对3款主流品牌的LED电视样机进行了较为详细的测试与评价。     

A weighted full-reference image quality assessment based on visual saliency

In full reference image quality assessment (IQA), the images without distortion are usually employed as reference, while the structures in both reference images and distorted images are ignored and all pixels are equally treated. In addition, the role of human visual system (HVS) is not taken account into subjective IQA metric. In this paper, a weighted full-reference image quality metric is proposed, where a weight imposed on each pixel indicates its importance in IQA. Furthermore, the weights can be estimated via visual saliency computation, which can approximate the subjective IQA via exploiting the HVS. In the experiments, the proposed metric is compared with several objective IQA metrics on LIVE release 2 and TID 2008 database. The results demonstrate that SROCC and PLCC of the proposed metric are 0.9647 and 0.9721, respectively,which are higher than other methods and it only takes 427.5 s, which is lower than that of most other methods.     

基于视觉显著性的立体显示灰阶串扰量化

为了更为合理地评价立体显示系统中的串扰现象,对传统灰阶串扰进行了加权并进一步单一数值量化。研究选取了大量涵盖各种类型和场景的立体图像,对每一灰阶转换组合出现的概率进行统计,同时提取立体图像视觉显著性,提出了平均灰阶转换显著性的概念。平均灰阶转换显著性以矩阵形式呈现,应用其对传统灰阶串扰矩阵加权后可获得更为合理的灰阶串扰。加权后的灰阶串扰矩阵求和后即可获得数值范围在0%~100%的单一数值串扰。提出的加权灰阶串扰及其数值单一化方法充分考虑了图像内容和人眼视觉关注机理的影响,得到的评价结果更加客观、合理。同时,研究结果对串扰现象的优化以及视觉舒适度的提高具有指导和推动作用。     

一种基于立体视觉显著性的多视点视频比特分配方法

针对多视点立体视频压缩编码,提出了一种基于立 体视觉显著性的比 特分配方法。研究综合利用多视点立体视频数据中场景的运动、深度以及深度边缘信息提取 人眼感兴趣区 域(ROI)的方法;然后根据ROI的划分结果优化区域比特分配。实验结果表 明,本文提出的算法能有效提 高ROI区域的编码性能,同时整体视频的率失真性能有一定程度的提高。     

Key-frame-based depth propagation for semi-automatic stereoscopic video conversion

In this paper, we propose a key-frame-based bi-directional depth propagation algorithm for semi-automatic 2D-to-3D stereoscopic video conversion. First, key-frames are identified from each video shot based on color motion-compensation errors to prevent high-motion content between any pair of consecutive key frames. Depths for key-frames are manually assigned or rendered by popular computer tools, and then bi-directionally propagated to non-key-frames there between. Our depth propagation algorithm is featured of a multi-pass error correcting procedure for each frame to prevent depth artifacts from being further propagated to adjacent frames. Our proposed algorithm is advantageous in solving the background occlusion/dis-occlusion problem that degrades the performances of traditional depth propagation algorithms. Experimental results show that our scheme is capable of achieving better results against three prior algorithms in view of the qualities of the estimated depth map (e.g., dis-occluded background and object boundaries) and the synthesized stereo views.     

色度对立体图像视觉舒适度影响的定量研究

随着立体成像技术的迅猛发展,对影响立体图像 观看舒适度的各种因素进行研究,已成为立体产品 迅速普及的关键技术之一。本文结合人眼视觉系统(HVS)特性,定量研究了色度因素 对观看双目立体图像舒适度的 影响。采用改进极限法,通过主观实验定义了立体图像舒适色度匹配图和差异图,并通过例 证验证了其普 适性。实验结果表明,舒适区域约占整个色度区域的48%,左右视图 色度不匹配的门限值会随着色调的不 同有所不同,且双目视图的色度值差异最大门限值的最大最小值分别为129.3°和46.1°,即左右视图色度 值的差异不能过大。根据两图给出的舒适色度范围,能够快速、准确地对测试立体图像进行 舒适与否的判 定,其正确率可达到90%以上,为立体图像舒适度评价及立体显示技 术的改进发展提供了依据。     

Multi-layer fusion network for blind stereoscopic 3D visual quality prediction

Stereoscopic 3D (S3D) visual quality prediction (VQP) is used to predict human perception of visual quality for S3D images accurately and automatically. Unlike that of 2D VQP, the quality prediction of S3D images is more difficult owing to complex binocular vision mechanisms. In this study, inspired by the binocular fusion and competition of the binocular visual system (BVS), we designed a blind deep visual quality predictor for S3D images. The proposed predictor is a multi-layer fusion network that fuses different levels of features. The left- and right-view sub-networks use the same structure and parameters. The weights and qualities for the left- and right-view patches of S3D images can be predicted. Furthermore, training patches with more saliency information can improve the accuracy of prediction results, which also make the predictor more robust. The LIVE 3D Phase I and II datasets were used to evaluate the proposed predictor. The results demonstrate that the performance of the proposed predictor surpasses most existing predictors on both asymmetrically and symmetrically distorted S3D images.     

基于深度感知和主观评价的影响立体图像舒适度的视差范围研究

立体图像的视差很大程度上决定了立体图像的视觉舒适度,本文从定量的 角度对影响立体图 像舒适度的视差范围进行了研究。首先对源立体图像采用像素平移方法得到测试立体图像集 ,通过大量的 主观实验得到测试立体图像的舒适度平均意见得分(MOS,mean o pinion score)值;其次,基于视觉显著性特点,采用自适应权重立体匹配方 法计算各区域的视差值,以测试各区域视差值对整幅立体图像舒适度的影响程度;最后通过 转换公式将舒 适立体图像的视差范围转换为视差角形式,以便将本文方法推广到其他类型的显示设备。实 验结果表明, 在实验所用显示设备中,满足舒适要求的像素平移量为－40～90(左 移40pixels至右移90pixels);当 立体图像中心显 著区域的水平视差在－0.3882°～0.851范围内时, 其舒适度在可接受范围内,而视差范 围为－0.2713°～0.781时舒适度达到最佳;最后将本文方法推广到常用显示设备,并给出了常用显示设 备的舒适视差范围,为立体图像和视频的制作提供了可靠依据。     

基于三维变换域频谱差的视频显著性检测算法

显著性区域检测是计算机视觉的重要课题,对视频质量评价和感知视频编码算法优化也至关重要.大多显著性检测算法不能权衡准确率和复杂度,限制了它们在视频预处理和实时处理中的应用.提出了一种基于三维变换域频谱差(3DTDSD)的快速视频显著性检测算法,分别以关键帧及其前一帧为中心建立一定数量图像帧的滑动窗,得到两组3D视频体,用傅里叶变换将两组视频变换到三维频域,两组三维数据之间的差值经过反变换得到显著性图,最后通过连通分析、阈值判断等得到显著区域.频域算法具有运算速度快的特点,实验对比和算法复杂度分析证明了该算法的有效性和快速性.     

Simultaneous 2D and 3D perception for stereoscopic displays based on polarized or active shutter glasses

Viewing stereoscopic 3D content is typically enabled either by using polarizing or active shutter glasses. In certain cases, some viewers may not wear viewing glasses and hence, it would be desirable to tune the stereoscopic 3D content so that it could be simultaneously watched with and without viewing glasses. In this paper we propose a video post-processing technique which enables good quality 3D and 2D perception of the same content. This is done through manipulation of one view by making it more similar to the other view to reduce the ghosting artifact perceived without viewing glasses while 3D perception is maintained. The proposed technique includes three steps: disparity selection, contrast adjustment, and low-pass filtering. The proposed approach was evaluated through an extensive series of subjective tests, which also revealed good adjustment parameters to suit viewing with and without viewing glasses with an acceptable 3D and 2D quality, respectively.     

视觉舒适度的理论模型及其图像研究

以韦伯—费昔勒定律为基础,建立了视觉舒适度的理论模型,利用拟合方法得出四种光环境下视觉舒适度的公式和图像。其结果表明：建立的视觉舒适度的数学模型能较好地反映视觉舒适度的客观规律,并能为节能照明设计提供定量的理论依据。     

面向3D呈现的有损和无损混合深度视频编码

针对多个深度视频流提出实时的压缩方法和评估 方案,采用一种有损无损混合的编码 方法,可以在图像质量和压缩率之间提供一种有效的平衡控制。行程编码(RLE)用于无损 压 缩,用来保存深度图像素的高位;像素低10bits保存在YUV图像的Y通道,直接使用×264编码。 实验表明,所提方法可以在平均时间小于12ms的前提下同时编解码 多个深度图。在实时传输 中,通过动态调节质量控制级别,可以获取4∶1到20∶ 1的压缩率。在压缩率10∶1的情况下,主观解码3D重构 效果与原始图几乎完全一致。     

3D saliency detection based on background detection

Unlike 2D saliency detection, 3D saliency detection can consider the effects of depth and binocular parallax. In this paper, we propose a 3D saliency detection approach based on background detection via depth information. With the aid of the synergism between a color image and the corresponding depth map, our approach can detect the distant background and surfaces with gradual changes in depth. We then use the detected background to predict the potential characteristics of the background regions that are occluded by foreground objects through polynomial fitting; this step imitates the human imagination/envisioning process. Finally, a saliency map is obtained based on the contrast between the foreground objects and the potential background. We compare our approach with 14 state-of-the-art saliency detection methods on three publicly available databases. The proposed model demonstrates good performance and succeeds in detecting and removing backgrounds and surfaces of gradually varying depth on all tested databases.     

基于视觉显著和CNN的花卉图像检索软件

花卉图像检索是图像检索领域的热门研究方向,高效、快速地检索数据库中的花卉图像一直是该方向的重点课题。为了检索花卉图像,文中设计了一个基于视觉显著模型和CNN的图像哈希算法,并根据此算法设计和开发出一个高效、快速的花卉图像检索软件。软件具有查询花卉类别、检索相似花卉、浏览花卉信息等功能。     

An enhanced image quality assessment by synergizing superpixels and visual saliency

Superpixel and saliency-based evaluation methods play important roles in full reference image quality assessment (FR IQA). However, we find that these methods have one complementary principle and three limitations: (1) the weighted maps of superpixel-based methods conflict with the perception of the human visual system; (2) saliency-based methods are inefficient in terms of the block distortion; (3) the general two-direction gradient extraction factor must be extended to be multidirectional. To address these limitations, we propose an enhanced image quality assessment by synergizing superpixels and visual saliency. Specifically, the calculation of a newly proposed framework involves three similarities and two strategies: the saliency, superpixel and multidirectional gradient similarities of the neighborhoods, and the saliency pooling strategy, the fusion strategy of these similarities. Theoretical analysis and experimental results show that the proposed method can effectively address the limitations noted above and outperform the existing methods.     

Toward an unsupervised blind stereoscopic 3D image quality assessment using joint spatial and frequency representations

Existing blind stereoscopic 3D (S3D) image quality assessment (IQA) metrics usually require supervised learning methods to predict S3D image quality, which limits their applicability in practice. In this paper, we propose an unsupervised blind S3D IQA metric that utilizes the joint spatial and frequency representations of visual perception. The metric proposed in this work was inspired by the binocular visual mechanism; furthermore, it is unsupervised and does not require subject-rated samples for training. To be more specific, first, the various binocular quality-aware features in spatial and frequency domains are extracted from the monocular and cyclopean views of natural S3D image patches. Subsequently, these features are utilized to establish a pristine multivariate Gaussian (MVG) model to characterize natural S3D image regularities. Finally, with the learned MVG model, the final quality score for a distorted S3D image can be yielded using a Bhattacharyya-like distance. Our experimental results illustrate that, compared to related existing metrics, the devised metric achieves competitive prediction performance.     

3D视频的质量损伤类型分析

首先介绍了3D视频通信系统和人类视觉系统,然后通过结构、颜色、运动和深度的4个视觉子系统,分析每个通信模块中各类立体质量损伤的现象、成因和解决办法.立体损伤与设备、表示方式、编解码算法以及主观感知等多重因素有关,了解质量损伤的类型有助于质量评价工作的进行,进而对系统的其他环节做出优化.     

3D visual experience oriented cross-layer optimized scalable texture plus depth based 3D video streaming over wireless networks

3D video streaming over the mobile Internet generally incurs the inferior 3D visual experience due to the time-varying characteristics of wireless channel. The conventional video streaming optimization methods generally neglect the harmony among different networking protocol layers. This paper proposes a cross-layer optimized texture plus depth based scalable 3D video streaming method to improve the expected 3D visual experience of the user by systematically considering the application layer texture-video/depth/FEC bit-rate allocation, MAC layer multi-channel allocation, and physical layer modulation and channel coding scheme (MCS) selection. In the cross-layer optimization, a networking-related 3D visual experience model which fuses the overlapped retinal view visual quality and depth sensation with mimicking human vision system is established to predict the 3D visual experience under the specific parameter configurations of different protocol layers. The efficiency and effectiveness of the proposed cross-layer optimized 3D video streaming method has been validated by subjective and objective experimental results.     

光栅3D显示视疲劳与视功能参数的关系

观看光栅3D显示的机理和人眼视觉生理存在矛盾 ,观看时间稍长将引起头痛、眼胀、复视等视疲 劳症状,选择恰当的生理指标对视疲劳进行客观评价对其预防和消除具有重要意义。本研究 探讨观看光栅 3D显示对双眼视功能参数的影响。通过视觉检查筛选出24名被试者,并随机分成两组进行 30分钟持续观 看光栅3D显示2D视频和3D视频的实验。在观看实验前后分别对被试者进行视功能参数—双眼 融合范围、 AC/A值和CFF值的测量,并完成视疲劳和视觉诱导晕动症主观问卷。实验验证了融合范围 、AC/A值和 CFF值都随着视疲劳和VIMS的加剧而降低。2D组观看前后融合范围无差异,3D组观看前 后融合范围有 显著区别,而2D组和3D组的AC/A值和CFF值在观看前后都有显著差异。3D观看前后融合 范围、AC/A 值、CFF值降低幅度均比2D更大。实验结果表明,3D比2D造成了更严重的视疲劳和VIMS ,而晕动症是导致观看光栅3D显示不舒适的主要原因。