Perceived quality measurement of stereoscopic 3D images based on sparse representation and binocular combination

Measurement of the perceived quality of stereoscopic three-dimensional (S3D) images has attracted an increasing amount of research interest in recent years. This paper proposes a S3D image quality measurement (IQM) metric based on sparse representation and binocular combination. The proposed method involves learning binocular and monocular dictionaries from a training database such that the sparse features of binocular combination can be expressed by a linear combination of a few selected basis feature vectors. Following this, scores for the similarity of these sparse features between reference and distorted S3D images are measured. Based on the observation that sparse features are invariant against weak degradations, similarity scores of the features of the gradient magnitude of binocular combination are then computed and used as a complementary feature. Finally, by using kernel-based support vector regression (SVR), these similarity scores are integrated into an overall quality value. Experimental results on three public S3D-IQM datasets show that in comparison with the relevant existing metrics, the devised metric attains significantly high consistency alignment with subjective quality assessment.     

Cognitive processes in complex tasks: introduction and discussion

This paper points out that the fusional limits of binocular stereoscopic viewing are an important factor, not only determining the reproduction range of stereoscopic images, but also the conformity between the stereoscopic display and stereoscopic vision. Experimental results showed that fusional limits increase in proportion to the field of view angle, and that they are affected not only by the size of the viewing target, but also by the effects that the environment has on the target. These findings explain the differences between binocular vision in real space and binocular vision in a stereoscopic display. Finally, the conditions under which observers arc able to view images of stereoscopic displays without excessive visual strain are discussed.     

Depth perception for moving images shown on a large LED display with an aperture grille

Abstract— A type of depth illusion created by the use of an aperture grille is reported. When viewing a moving target through multiple slits, a movement with depth is perceived, which was originally reported (see Ref. 11). The binocular delay is considered to cause a virtual disparity between both perspective images with apparent movement. By using an LED display with an aperture grille as a stereoscopic display, perceived distance caused by a binocular delay has been measured. The measured distance is compared with the perceived distance for stereoscopic still images shown on a stereoscopic LED panel. The comparison supported that the binocular delay is converted into binocular disparity. Furthermore, pair‐comparison tests were conducted to investigate depth impressions. It was found that use of an aperture grille improves depth impression for a movie that was taken with a laterally moving camera.     

考虑双目竞争视觉现象的非对称失真立体图像质量评价方法

目的 现有方法存在特征提取时间过长、非对称失真图像预测准确性不高的问题,同时少有工作对非对称失真与对称失真立体图像的分类进行研究,为此提出了基于双目竞争的非对称失真立体图像质量评价方法。方法 依据双目竞争的视觉现象,利用非对称失真立体图像两个视点的图像质量衰减程度的不同,生成单目图像特征的融合系数,融合从左右视点图像中提取的灰度空间特征与HSV （hue-saturation-value）彩色空间特征。同时,量化两个视点图像在结构、信息量和质量衰减程度等多方面的差异,获得双目差异特征。并且将双目融合特征与双目差异特征级联为一个描述能力更强的立体图像质量感知特征向量,训练基于支持向量回归的特征—质量映射模型。此外,还利用双目差异特征训练基于支持向量分类模型的对称失真与非对称失真立体图像分类模型。结果 本文提出的质量预测模型在4个数据库上的SROCC （Spearman rank order correlation coefficient）和PLCC （Pearson linear correlation coefficient）均达到0.95以上,在3个非对称失真数据库上的均方根误差（root of mean square error,RMSE）取值均优于对比算法。在LIVE-II（LIVE 3D image quality database phase II）、IVC-I（Waterloo-IVC 3D image qualityassessment database phase I）和IVC-II （Waterloo-IVC 3D image quality assessment database phase II）这3个非对称失真立体图像测试数据库上的失真类型分类测试中,对称失真立体图像的分类准确率分别为89.91%、94.76%和98.97%,非对称失真立体图像的分类准确率分别为95.46%,92.64%和96.22%。结论 本文方法依据双目竞争的视觉现象融合左右视点图像的质量感知特征用于立体图像质量预测,能够提升非对称失真立体图像的评价准确性和鲁棒性。所提取双目差异性特征还能够用于将对称失真与非对称失真立体图像进行有效分类,分类准确性高。     

基于单目和双目视觉信息的全参考立体图像质量评价模型

在立体图像质量评价领域,有效地模拟人类视觉系统对图像质量进行评价具有重要意义,考虑到人眼的视觉感知特性,基于单目和双目视觉信息构建一种立体图像质量评价模型MB-FR-SIQA。采用基于结构相似性的立体视差算法得到参考和失真立体图像的视差矩阵,结合Gabor能量响应图、显著性图和视差矩阵生成中间视图,并优化左右眼加权系数计算方法,以提高生成中间视图的准确性。分别利用单目图像和中间视图提取单目和双目视觉信息,计算单目质量分数和双目质量分数,并融合得到立体图像的质量分数,达到评价立体图像质量的目的。实验结果表明,MB-FR-SIQA模型在LIVE-I数据库上具有较高的预测精度,其斯皮尔曼等级相关系数、皮尔森线性相关系数、均方根误差分别为0.945、0.951、5.318,且预测的质量分数符合人类主观评估。     

基于空域自然场景统计的无参考立体图像质量评价模型

针对现有的评价方法大都将图像变换到不同的坐标域问题,提出一种基于空域自然场景统计(NSS)的通用型无参考立体图像质量评价模型。在评价中为了更好地结合人类双目视觉特性, 将左右图像融合成一幅独眼图;评价模型首先统计独眼图归一化亮度(CMSCN)系数分布规律,进而对独眼图提取空域自然场景统计特征;其次,统计视差图归一化亮度(DMSCN)系数的分布规律,并对用光流法得到的视差图提取同样的特征;最后,通过支持向量回归(SVR)建立立体图像特征信息与主观评价值(DMOS)之间的关系,从而预测得到图像质量的客观评价值。实验结果表明,该评价模型对立体数据测试库进行评价,其Pearson线性相关系数(PLCC)和Spearman等级相关系数(SROCC)值均在0.94以上;对于非对称立体图像库,PLCC和SROCC值分别接近0.91和0.93。该模型能够很好地预测人眼对立体图像的主观感知。     

Blind stereo image quality assessment inspired by brain sensory-motor fusion

The use of 3D and stereo imaging is rapidly increasing. Compression, transmission, and processing could degrade the quality of stereo images. Quality assessment of such images is different than their 2D counterparts. Metrics, which represent 3D perception by human visual system (HVS), are expected to assess stereoscopic quality more accurately. In this paper, inspired by brain sensory/motor fusion process, by combining the right and left images, we form two synthesized images. Effects of different structural distortions on statistical distributions of the synthesized images are analyzed. Based on the observed statistical changes, features are extracted from these synthesized images that can reveal type and severity of distortions. Then, we propose a stacked neural network model to learn the extracted features and accurately predict the quality of stereo images. This model is tested on 3D images of popular databases. Experimental results show the superiority of this method over state-of-the-art stereo image quality assessment approaches.     

The effect of depth map up-sampling on the overall quality of stereopairs

The quality of depth maps affects the quality of generated 3D content. Practically, the depth maps often have lower resolution than that of color images, thus, Depth map Up-sampling (DU) is needed in various 3D applications. DU can yield specific artifacts which can degrade the quality of depth maps as well as constructed stereoscopic (color plus depth map) images. This paper investigates the effect of DU on 3D perception. The depth maps were up-sampled using seven approaches and the quality of stereoscopic images obtained from up-sampled depth maps was estimated through subjective and objective tests. The objective quality prediction was performed using a depth map quality assessment framework. The method is able to predict the quality of stereoscopic images through evaluation of their corresponding up-sampled depth maps using 2D Image Quality Metrics (IQMs). In order to improve the quality estimation, the framework selects the 2D IQMs with highest correlation to subjective test. Furthermore, motivated by previous researches on multiple metrics combination, a new metric fusion method is proposed. Experimental results show that the combined metric delivers higher performance than single metrics in 3D quality prediction.     

Evaluation of stereoscopic picture quality with CFF

The method of binocular stereoscopic imaging has attracted attention as a simple and technically feasible means of three-dimensional (3-D) display. It has been said, however, that the binocular picture system is more fatiguing to view than a conventional two-dimensional picture, such as the current television picture. The main objective of this paper is to evaluate quantitatively fatigue caused by binocular stereoscopic picture viewing, to clarify the basic characteristics of this type of picture. Various measurement methods of fatigue have been proposed, and that using the critical flicker frequency (CFF) has found wide use in measurement of mental fatigue and is easy to make. Actual measurement in this study used the CFF value obtained in a descending series of the method of limits, normalized to the value at the start of measurement for each observer, and the CFFs of all observers then averaged. It was found that the measurements produced stable and highly accurate results. CFFs of the observers of a current television picture and a binocular stereoscopic television picture were measured. It was found that the conventional television did not cause a statistically significant decrease in CFF—even over 1?h of viewing—but that the stereoscopic television did cause a significant CFF decrease within 30?min. It was also found that the CFF decreases are related to a subjective feeling of fatigue. From these results, it was concluded that CFF is an effective measure for objectively measuring the fatigue of television observers, including observers of binocular stereoscopic television. Also, the binocular stereoscopic television produced higher fatigue in observers than did the conventional television picture, but it was found that with respect to CFF viewing of around 30?min was allowable.     

Content-based image quality metric using similarity measure of moment vectors

In this paper, the similarity of moment vectors between the test and the reference image blocks together with the result from the block classification are used in the formulation of an image quality metric (IQM). First, the reference and the test images are divided into non-overlapping 8×8 blocks and transformed into moment domain using Discrete Tchebichef Transform. The moment features are then used in two operations: the local quality index calculation and the image content (block) classification. The local quality index is obtained from the similarity measure of moment vectors between the reference and the test image blocks. Next, the content of each reference image block is classified into three types: “plain”, “edge” and “texture”, based on its moment energy level and moment energy distribution. The local quality indices obtained from all the image blocks are then averaged based on the block types to obtain three mean quality scores for each test image. The performance of these three mean quality scores and their combinations are studied using the LIVE database. The results show that the performance of the metric is significantly improved by combining the mean quality scores from the edge and texture image region. The best combination (the proposed metric) is then compared with five other IQMs using the LIVE database and four other independent databases. The results show that the proposed metric performs comparatively well for all the databases.     

A new depth image quality metric using a pair of color and depth images

Typical depth quality metrics require the ground truth depth image or stereoscopic color image pair, which are not always available in many practical applications. In this paper, we propose a new depth image quality metric which demands only a single pair of color and depth images. Our observations reveal that the depth distortion is strongly related to the local image characteristics, which in turn leads us to formulate a new distortion assessment method for the edge and non-edge pixels in the depth image. The local depth distortion is adaptively weighted using the Gabor filtered color image and added up to the global depth image quality metric. The experimental results show that the proposed metric closely approximates the depth quality metrics that use the ground truth depth or stereo color image pair.     

基于视差图和复数轮廓波变换的无参考立体图像质量评价

现有的2D图像质量评价方法并不能很好地应用于立体图像质量评价中。为了有效评价不同失真立体图像的质量,提出了一种基于视差图和复数轮廓波变换的无参考图像质量评价方法。首先提取了能够反映3D信息的视差图,然后对左右失真图像和视差图进行复数轮廓波变换,计算能量和能量差特征,最后通过支持向量回归SVR模型训练学习,预测图像质量分数。实验结果表明,此方法优于当前文献报道的立体图像质量评价方法。     

Perceptual quality assessment of stereoscopic images based on local and global visual characteristics

The quality assessment of stereoscopic images has attracted considerable attention and become an important issue in 3D multimedia applications. The 3D image quality assessment (IQA) encounters many challenges and simple extension of the 2D quality metrics to the 3D case is not satisfying. In this paper, we propose a new perceptual quality assessment scheme for stereoscopic 3D images by considering the local and global visual characteristics. The design of this scheme is motivated by studies on the perception of distorted stereoscopic images. To be more specific, after the log-Gabor filter processing, the local amplitude and phase from the left and right views of the reference and distorted 3D images are utilized as features in local quality evaluation. Meanwhile, the global structure changes of the left and right views are also incorporated into the final quality pooling. The overall 3D quality score is obtained by combining the local and global quality indexes together. The effectiveness of the designed metric is verified on publicly available 3D image quality assessment databases. Experimental results show that the proposed scheme exhibits better performance than other related algorithms in terms of consistency with subjective assessment of stereoscopic 3D images.

     

No-reference stereoscopic image quality assessment using quaternion wavelet transform and heterogeneous ensemble learning

As the demand for high-quality stereo images has grown in recent years, stereoscopic image quality assessment (SIQA) has become an important research area in modern image processing technology.In this paper, we propose a no-reference stereoscopic image quality assessment (NR-SIQA) model using heterogeneous ensemble learning ‘quality-aware’ features from luminance image, chrominance image, disparity and cyclopean images via quaternion wavelet transform (QWT). Firstly, luminance image and chrominance image are generated by CIELAB color space as monocular perception, and the novel disparity and cyclopean images are utilized to complement with monocular information. Then, a number of ‘quality-aware’ features in the quaternion wavelet domain are discovered, including entropy, texture features, energy features, energy differences features and MSCN coefficients of high frequency sub-band. Finally, a heterogeneous ensemble model via support vector regression (SVR) & extreme learning machine (ELM) & random forest (RF) is proposed to predict quality score, and bootstrap sampling and rotated feature space are used to increase the diversity of data distribution. Comparing with the state-of-the-art NR-SIQA models, experimental results on four public databases prove the accuracy and robustness of the proposed model.     

基于双树复小波的无参考立体图像质量评价

随着3D技术的不断发展，立体图像的使用领域越来越广泛，同时人们对图像的清晰度要求越来越高，因此，立体图像的质量评价成为关注点，基于此，提出了一种基于双树复小波变换的立体图像质量评价算法。使用双树复小波变换对立体图像的左、右视图进行处理，生成纹理结构图像，且根据最小能量误差的原理，获取左右视图的视差图；对纹理结构图像和视差图提取非对称广义高斯分布模型的参数、梯度幅值、相对梯度方向方差和奇异值曲线与坐标轴的面积等特征；使用AdaBoosting BP神经网络，进行训练和预测立体图像的质量得分。在LIVE立体图像数据库上的实验结果表明，新方法预测得分与主观得分有较好的一致性，获得了比较好的实验结果。     

Quality assessment of 3D synthesized images based on structural and textural distortion

Depth image based rendering (DIBR) is a popular technique for rendering virtual 3D views in stereoscopic and autostereoscopic displays. The quality of DIBR-synthesized images may decrease due to various factors, e.g., imprecise depth maps, poor rendering techniques, inaccurate camera parameters. The quality of synthesized images is important as it directly affects the overall user experience. Therefore, the need arises for designing algorithms to estimate the quality of the DIBR-synthesized images. The existing 2D image quality assessment metrics are found to be insufficient for 3D view quality estimation because the 3D views not only contain color information but also make use of disparity to achieve the real depth sensation. In this paper, we present a new algorithm for evaluating the quality of DIBR generated images in the absence of the original references. The human visual system is sensitive to structural information; any deg radation in structure or edges affects the visual quality of the image and is easily noticeable for humans. In the proposed metric, we estimate the quality of the synthesized view by capturing the structural and textural distortion in the warped view. The structural and textural information from the input and the synthesized images is estimated and used to calculate the image quality. The performance of the proposed quality metric is evaluated on the IRCCyN IVC DIBR images dataset. Experimental evaluations show that the proposed metric outperforms the existing 2D and 3D image quality metrics by achieving a high correlation with the subjective ratings.

     

Subjective evaluation of visual fatigue due to misalignment of motion and still images in a stereoscopic display

Abstract— In this paper, the relative influences of misalignment such as cross‐talk, vertical shift, and motion blur on visual fatigue by using a binocular stereoscopic display has been verified. Experiments were conducted for two cases: a still image and a motion image. They were evaluated by using the simulator sickness questionnaire (SSQ). By changing the disparity angle, cross‐talk, and vertical shift in the still‐image experiment, it was found that the SSQ score of each parameter increased as the amount of each factor increased. With a two‐sample t‐test between the presence and absence of each factor, a significant difference was found for the case of a more than 36‐arcmin disparity, more than 20% cross‐talk, and more than 10% vertical shift. In the motion‐image experiment, in which motion speed and the misalignment were varied, it was found that movement of the disparity angle caused much more visual fatigue in comparison with the misalignment factors, which were cross‐talk and vertical shift. In contrast, motion images in addition to the cross‐talk and/or vertical shift had a slight but unnoticeable relationship to an increase in visual fatigue. Therefore, it was concluded that vertical shift dominated the evaluation for still images, and the movement itself dominated the evaluation for motion images. The results suggest that it is necessary to evaluate visual fatigue according to the representing case, still or motion images, of a 3‐D stereoscopic display.     

Quality assessment for virtual reality technology based on real scene

Virtual reality technology is a new display technology, which provides users with real viewing experience. As known, most of the virtual reality display through stereoscopic images. However, image quality will be influenced by the collection, storage and transmission process. If the stereoscopic image quality in the virtual reality technology is seriously damaged, the user will feel uncomfortable, and this can even cause healthy problems. In this paper, we establish a set of accurate and effective evaluations for the virtual reality. In the preprocessing, we segment the original reference and distorted image into binocular regions and monocular regions. Then, the Information-weighted SSIM (IW-SSIM) or Information-weighted PSNR (IW-PSNR) values over the monocular regions are applied to obtain the IW-score. At the same time, the Stereo-weighted-SSIM (SW-SSIM) or Stereo-weighted-PSNR (SW-PSNR) can be used to calculate the SW-score. Finally, we pool the stereoscopic images score by combing the IW-score and SW-score. Experiments show that our method is very consistent with human subjective judgment standard in the evaluation of virtual reality technology.

     

Disparity-based just-noticeable-difference model for perceptual stereoscopic video coding using depth of focus blur effect

Human 3D perception provides an important clue to the removal of redundancy in stereoscopic 3D (S3D) videos. Because objects outside the binocular fusion limit cannot be fused on retina, the human visual system (HVS) makes them blur according to the depth-of-focus (DOF) effect to increase the binocular fusion limit and suppress diplopia, i.e. double vision. Based on human depth perception, we propose a disparity-based just-noticeable-difference model (DJND) to save bit-rate and improve visual comfort in S3D videos. We combine the DOF blur effect with conventional JND models in the pixel domain into DJND. Firstly, we use disparity information to get the average disparity value of each block. Then, we integrate the DOF blur effect into luminance JND (LJND) by a selective low pass Gaussian filter to minimize the visual stimulus in S3D videos. Finally, we incorporate disparity information into the filtered JND models to obtain DJND. Experimental results demonstrate that the proposed method successfully improves both image quality and visual comfort in viewing S3D videos without increasing the bit-rate.     

基于眼优势的非对称失真立体图像质量评价

针对现有立体图像质量评价算法对非对称失真立体图像的评价准确性及执行效率较低的问题,提出一种基于眼优势的非对称失真立体图像质量评价算法.首先采用梯度幅值响应来模拟左右眼输入的刺激强度,并根据人类视觉系统的眼优势原理分别以左和右视点图像作为主视图合成两幅融合图像;其次,利用旋转不变统一局部二值模式直方图、皮尔逊线性相关系数以及非对称广义高斯模型,获取左右融合图像以及左右梯度幅值响应图像中的多种能够反映立体图像质量好坏的特征;最后,利用自适应增强的支持向量回归模型将感知特征向量映射为图像质量值.在四个基准测试数据库上的实验结果表明:本文所提出算法大幅提升了非对称失真立体图像的评价准确性,且具有较高的执行效率.这些优势说明本文算法所提取的特征描述能力更强,质量映射模型的稳定性更好.