基于双目特征联合的无参考立体图像质量评价

通过模拟人类视觉系统(HVS)的双目视觉行为,提 出一种基于双目特征联合的无参考立 体图像质量评价(NR-SIQA)方法。首先分析立体视觉感知中的双目联合行为,提出 可应用于立体图像质量预 测的双目联合模型;然后采用学习和统计分析的方法,分别提取局部和全局特征并联合作 为感知特征; 最后采用机器学习算法,建立特征和质量的关系模型,并结合基于特征的双目联合模型预测 立体图像质量。实验结果表明,本文方法在对称立体图像库上的Pearson线性相关系数(PLCC)和Spearman等级系数(SRCC)高于0.93,在非对称库上高于0.87,优 于现有评价方法。     

Quality assessment of multiply and singly distorted stereoscopic images via adaptive construction of cyclopean views

A challenging problem confronted when designing a blind/no-reference (NR) stereoscopic image quality assessment (SIQA) algorithm is to simulate the quality assessment (QA) behavior of the human visual system (HVS) during binocular vision. An effective way to solve this problem is to estimate the quality of the merged single view created in the human brain which is also referred to as the cyclopean image. However, due to the difficulty in modeling the binocular fusion and rivalry properties of the HVS, obtaining effective cyclopean images for QA is non-trivial, and consequently previous NR SIQA algorithms either require the MOS/DMOS values of the distorted 3D images for training or ignore the quality analysis of the merged cyclopean view. In this paper, we focus on (1) constructing accurate and appropriate cyclopean views for QA of stereoscopic images by adaptively analyzing the distortion information of two monocular views, and (2) training NR SIQA models without requiring the assistance of the MOS/DMOS values in existing databases. Accordingly, we present an effective opinion-unaware SIQA algorithm called MUSIQUE-3D, which blindly assesses the quality of multiply and singly distorted stereoscopic images by analyzing quality degradations of both monocular and cyclopean views. The monocular view quality is estimated by an extended version of the MUSIQUE algorithm, and the cyclopean view quality is computed from the distortion parameter values predicted by a two-layer classification-regression model trained on a large 3D image dataset. Tests on various 3D image databases demonstrate the superiority of our method as compared with other state-of-the-art SIQA algorithms.     

Stereoscopic image quality assessment considering visual mechanism and multi-loss constraints

In this paper, a convolutional neural network (CNN) with multi-loss constraints is designed for stereoscopic image quality assessment (SIQA). A stereoscopic image not only contains monocular information, but also provides binocular information which is as identically crucial as the former. So we take the image patches of left-view images, right-view images and the difference images as the inputs of the network to utilize monocular information and binocular information. Moreover, we propose a method to obtain proxy label of each image patch. It preserves the quality difference between different regions and views. In addition, the multiple loss functions with adaptive loss weights are introduced in the network, which consider both local features and global features and constrain the feature learning from multiple perspectives. And the adaptive loss weights also make the multi-loss CNN more flexible. The experimental results on four public SIQA databases show that the proposed method is superior to other existing SIQA methods with state-of-the-art performance.     

No-reference stereoscopic image quality evaluator based on human visual characteristics and relative gradient orientation

Stereoscopic image quality assessment (SIQA) is of great significance to the development of modern three-dimensional (3D) display technology. In this work, by further mining the relationship between visual features and stereoscopic image quality perception, we build a new no-reference SIQA model, which combines the monocular and binocular features. Statistical quality-aware structural features from relative gradient orientation (RGO) map and texture features from the histogram of the weighted local binary pattern (LBP) in the texture image (TLBP) are not only extracted from both monocular view, but also extracted from binocular views to predict binocular quality perception. Meanwhile, the color statistical features ignored by most models and the binocularity feature is extracted to complement the monocular features and the above binocular features, respectively. Finally, all the extracted features and subjective scores are used to predict the objective quality score through the support vector regression (SVR) model. Experiments on four popular stereoscopic image databases show that the proposed model achieves high consistency with subjective assessment, and the performance of the model is very competitive with the latest models.     

Two-stream interactive network based on local and global information for No-Reference Stereoscopic Image Quality Assessment

Nowadays, stereoscopic image quality assessment (SIQA) based on convolutional neural network (CNN) has become the mainstream model of image quality assessment (IQA). Compared with the two-dimensional quality evaluation model, stereoscopic image quality evaluation is more challenging due to the effects of depth and parallax information. In this paper, we propose a two-stream interactive network model to perform quality evaluation, which can well simulate the process of human stereo visual perception. Meanwhile, we enhance the extraction of local and global features of images by asymmetric convolution kernel and interactive sub-networks of inter-layers, respectively, which can further optimize our network model. Our proposed algorithm was evaluated on four public databases. The final experimental results show that our proposed algorithm exhibits good performance not only on the whole database but also on each single distortion type.     

基于稀疏表示的立体图像客观质量评价方法

提出了一种基于稀疏表示的立体图像质量评价方法 ,分为训练和测试两个部分。在训练部 分,通过训练不同频带的立体图像获得立体图像的稀疏字典;在测试部分,根据稀疏字典计 算得到立体图 像的稀疏特征,定义了稀疏特征相似度衡量原始和失真图像信息的差异,并根据稀疏字典计 算了频带增益和左右视点的融合权值,最后融合稀疏特征相似度作为立体图像质量的 客观评价值。在立体图像测试库上的实验结果表明,本文方法的评价结果与主观评价结果有 较好的相关性,符合人类视觉系统的感知。     

No-reference Stereoscopic Image Quality Assessment Using Binocular Self-similarity and Deep Neural Network

Quality assessment of three-dimensional (3D) images is more challenging than that of 2D images. The quality of 3D visual experience is one of the most challenging areas of human binocular perception and is affected by multiple factors such as asymmetric stereo image/video compression, depth perception, visual discomfort, and single view quality. In this paper, we propose a new no-reference quality assessment method for stereoscopic images based on Binocular Self-similarity (BS) and Deep Neural Networks (DNN). To be more specific, a BS index is defined and computed according to binocular rivalry and suppression based on the depth image-based rendering technique. Then, a DNN is trained in an opinion unaware way to predict local quality. Binocular integration (BI) index is calculated by using the trained DNN, accounting for binocular integration behaviors. Finally, the final quality score of stereoscopic image is obtained by combining the BS and BI indexes together. Experimental results on four public 3D image quality assessment databases demonstrate that compared with existing methods, the proposed method can achieve high consistency with subjective perception on stereoscopic images with both symmetric and asymmetric distortions.     

Blind quality assessment for multiply distorted stereoscopic images towards IoT-based 3D capture systems

Empowered by 5G mobile communication networks, multimedia processing has been considered as a very promising application of Internet-of-Things (IoT). Stereoscopic image quality assessment (SIQA), as an important part of 3D capture system, can be embedded in the cloud or fog servers to automatically monitor the perceptual quality of the collected stereoscopic images. In this paper, a novel blind image quality assessment method towards IoT-based 3D capture systems is developed for multiply-distorted stereoscopic images (MDSIs), in which five complementary channels, including left view, right view, cyclopean map, summation map and difference map, are jointly considered in dictionary learning for characterizing the monocular receptive field (MRF) and binocular receptive field (BRF) properties of the visual cortex in response to MDSIs. Additionally, the high order statistics scheme is adopted by utilizing the statistical differences between the codebook and images to ensure the stable and robust quality prediction performance for MDSIs. The proposed method shows competitive prediction performances on four benchmark databases compared with the existing SIQA metrics.     

Stereo image quality assessment considering the difference of statistical feature in early visual pathway

Human visual theory is closely related to stereo image quality assessment (SIQA), which determines whether the evaluation results of SIQA method can keep good consistency with subjective perception. Many SIQA methods are not fully based on human visual theory, so there is still room for improvement. The research on the visual system tends to the dorsal and ventral pathways, which ignores the information differences in the early visual pathways. It is worth noting that the ON and OFF receptive fields in retinal ganglion cells (RGCs) respond asymmetrically to the statistical features of images. Inspired by this, in this paper, we propose an SIQA method based on monocular and binocular visual features, which takes into account the difference of ON and OFF response features in early visual pathways. Moreover, the different information interaction mechanisms of visual cortex are used to fuse the response maps information of left and right images. Final, monocular and binocular features are extracted and sent to support vector regression (SVR) for quality prediction. Experimental results show that the proposed method is superior to several mainstream SIQA metrics on four publicly available stereo image databases.     

一种基于最小可察觉失真的立体图像质量客观评价方法

立体图像质量是评价立体视频系统性能的有效途径,而如何利用人类视觉特性对立体图像质量进行有效的评价是目前的研究难点。本文通过分析最小可察觉失真(JND,just noticeable distortion)视觉感知模型,并结合反映图像结构信息的奇异值矢量,提出了一种基于JND的立体图像质量客观评价方法。评价方法由图像质量评价和深度感知评价两部分组成,首先提取反映图像质量和深度感知的特征信息作为立体图像特征信息,然后根据立体图像的不同失真类型情况对其特征进行融合,通过支持向量回归(SVR,support vector Regression)预测得出立体图像质量的客观评价值。实验结果表明,采用本文提出的客观评价方法对立体数据测试库进行评价,在不同失真类型或混合失真评价结果中,Pearson线性相关系数(CC)值均在0.94以上,Spearman等级相关系数(SROCC)值均在0.92以上,符合人眼视觉特性,能够很好地预测人眼对立体图像的主观感知。     

基于双目能量响应的无参考立体图像质量评价

为了实现对不同失真类型立体图像的质量评价,提出了一种基于双目能量响应的无参考立体图像质量评价(NR-IAQ)方法。首先,通过对各失真图像进行Gabor滤波,提取出不同频率、不同方向、不同视差响应下的局部特征矢量,作为立体图像特征信息;然后,利用支持向量回归(SVR)建立立体图像特征与主观评价值的关系,从而预测得到立体图像质量的客观评价值。实验结果表明,对于NBU-3D测试库,Pearson线性相关系数值在0.92以上,Spearman等级相关系数值在0.93以上;对于LIVE-3D测试库,Pearson线性相关系数值在0.96以上,Spearman等级相关系数值在0.96以上;与现有的全参考(FR)和(NR)质量评价方法相比,本方法得到的客观评价值与主观评价结果有较好的相关性,更加符合人眼视觉系统。     

Blind S3D image quality prediction using classical and non-classical receptive field models

We develop a novel no-reference image quality assessment model for stereoscopic 3D (S3D) images that is inspired by functional receptive field models of perceptual mechanisms in primary visual cortex (V1). The approach is called the Blind S3D Integrated Quality Evaluator (BSIQE). BSIQE simulates monocular and binocular responses to stereo views using channel separation and weighted multi-channel combination models. Binocular responses are modeled as the fusion of the two channels using a weighted multi-channel combination. The responses to stereoscopic image content of both classical and non-classical anisotropic receptive fields are then modeled based on a determination of the relative importance of the receptive field responses. In the last stage of feature extraction, we deploy a simple and efficient way of decorrelating the picture data. We extract local binary pattern (LBP) statistical features from the computed receptive field responses, and use them to train a regressor to predict the perceptual quality of stereoscopic images. We carefully evaluate BSIQE on four public-domain 3D image quality databases, and find that it is statistically superior to all compared 2D and 3D IQA algorithms. BSIQE exhibits good performance across the datasets suggesting that it is general, and it has relatively low complexity.     

No-reference stereoscopic images quality assessment method based on monocular superpixel visual features and binocular visual features

No-reference quality assessment of images has received considerable attention. However, the accuracy of such assessment remains questionable because of its weak biological basis. In this paper, we propose a novel quality assessment model based on the superpixel index and biological binocular mechanisms. The technical contributions of our model are the introduction of local monocular superpixel features and three global binocular visual features. We utilize monocular superpixel segmentation to extract two types of entropies as the local visual features for accurate quality-aware feature extraction. In addition, natural scene statistics features are extracted from the binocular visual information to complement the local monocular features and quantify the naturalness of the stereoscopic images. Finally, a regression model is learned to evaluate the quality of the stereoscopic images. Experimental results from three popular databases demonstrate that the proposed model has a more reliable performance than earlier models in terms of prediction accuracy and generalizability.     

基于视差空间图的立体图像质量客观评价方法

立体图像质量评价是评价立体视频系统性能的有 效途径,而如何利用人类视觉特性对立体图像质量 进行有效评价是目前的研究难点。本文提出了一种基于视差空间图(DSI) 的立体图像质量客观评价方法。首先, 分别构造原始立体图像和失真立体图像的DSI图;然后,通过三维离散余弦变换(3D-DCT)提取出反映图像质量 和深度感知的特征信息,并采用主成分分析(PCA)进行特征降维,形成立体图像特征信息; 最后,通过支持向量 回归(SVR)建立立体图像特征与主观评价值的关系,从而预测得到立体图像质量的客观评价 值。实验表明, 对于对称立体图像库,Pearson线性相关系数(PLCC)和Spe arman等级相关系数(SROCC)值均达到0.94以上;对于非 对称立体图像库,PLCC和SROCC值分别达到0.94以上。结果表明,本文方法能够很好地预测人眼对立体图像的主观感 知。     

结合空间域与变换域特征提取的盲立体图像质量评价

针对立体图像质量预测准确性不足的问题,该文提出了一种结合空间域和变换域提取质量感知特征的无参考立体图像质量评价模型。在空间域和变换域分别提取输入的左、右视图的自然场景统计特征,并在变换域提取合成独眼图的自然场景统计特征,然后将其输入到支持向量回归(SVR)中,训练从特征域到质量分数域的预测模型,并以此建立SIQA客观质量评价模型。在4个公开的立体图像数据库上与一些主流的立体图像质量评价算法进行对比,以在LIVE 3D Phase I图像库中的性能测试为例,Spearman秩相关系数、皮尔逊线性相关系数和均方根误差分别达到0.967,0.946和5.603,验证了所提算法的有效性。     

A ParaBoost stereoscopic image quality assessment (PBSIQA) system

The problem of stereoscopic image quality assessment, which finds applications in 3D visual content delivery such as 3DTV, is investigated in this work. Specifically, we propose a new ParaBoost (parallel-boosting) stereoscopic image quality assessment (PBSIQA) system. The system consists of two stages. In the first stage, various distortions are classified into a few types, and individual quality scorers targeting at a specific distortion type are developed. These scorers offer complementary performance in face of a database consisting of heterogeneous distortion types. In the second stage, scores from multiple quality scorers are fused to achieve the best overall performance, where the fuser is designed based on the parallel boosting idea borrowed from machine learning. Extensive experimental results are conducted to compare the performance of the proposed PBSIQA system with those of existing stereo image quality assessment (SIQA) metrics. The developed quality metric can serve as an objective function to optimize the performance of a 3D content delivery system.     

基于未匹配子带合成系数统计特性的立体图像质量评价算法

针对经典立体图像质量评价算法存在评价准确性 低以及特征提取耗时较长的问题, 提出一种基于未匹配子带合成系数统计特性的立体图像质量评价算法。首先,利用可控金字 塔对左右视点图像进行多尺度、多方向的小波分解,并将左右视点图像在相同尺度、相同方 向上未经视差图匹配的小波子带系数合成为子带合成系数。其次,提取小波子带合成系数中 的统计分布特征,相同尺度相邻方向小波子带合成系数之间的相关性特征,以及相同方向相 邻尺度子带合成系数之间的相关性特征。最后,利用所提取特征训练经典的支持向量回归模 型,预测图像质量。在LIVE 3D和Waterloo IVC 3D数据库上的实验结果表明,与主流立体 图像质量评价算法相比,本文算法在预测对称和非对称失真立体图像质量时都获得了更高的 评价准确性。同时,由于子带合成系数的生成无需根据视差图进行匹配,算法执行效率高。     

基于双目能量模型的立体图像质量客观评价方法

立体图像的深度感知取决于双目差距,基础视觉皮层的差距调谐细胞在感知立体深度的过程中起着关键作用.单只眼睛感知的图像信息被单目简单细胞接收之后传入双目简单细胞进行处理,继而被送入复杂细胞进行综合,得到基础视觉皮层对于一幅立体图像的能量响应.用数学模型来仿真简单的以及复杂的细胞对于双目差距的响应,并通过双目能量模型的计算达到评价彩色立体图像的目的.实验结果表明,双目能量模型的评价结果与立体图像的主观评价值具有较高的一致性.     

A new objective stereoscopic image assessment model based on stereoscopic perception

In order to establish a stereoscopic image quality assessment method which is consistent with human visual perception, we propose an objective stereoscopic image quality assessment method. It takes into account the strong correlation and high degree of structural between pixels of image. This method contains two models. One is the quality synthetic assessment of left-right view images, which is based on human visual characteristics, we use the Singular Value Decomposition （SVD） that can represent the degree of the distortion, and combine the qualities of left and right images by the characteristics of binocular superposition. The other model is stereoscopic perception quality as- sessment, due to strong stability of image＇s singular value characteristics, we calculate the distance of the singular values and structural characteristic similarity of the absolute difference maps, and utilize the statistical value of the global error to evaluate stereoscopic perception. Finally, we combine two models to describe the stereoscopic image quality. Experimental results show that the correlation coefficients of the proposed assessment method and the human subjective perception are above 0.93, and the mean square errors are all less than 6.2, under JPEG, JP2K compression, Gaussian blurring, Gaussian white noise, H.264 coding distortion, and hybrid cross distortion. It indicates that the proposed stereoscopic objective method is consistent with human visual properties and also of availability.     

基于三维感知的立体虚拟视点图像质量评价方法

为了评价立体虚拟视点图像的质量,提出了一种基 于三维感知的客观评价方法。综合考虑了立体虚拟视点图像两大最主要失真类型:单视点绘 制失真和立体视点不匹配失真。针对单视点绘制失真,先提取 当前视点失真图与无失真图的差异性区域,再针对该差异性区域计算平均结构相似度(MSSIM ),最后将左 右视点平均池化作为单目纹理特征值;针对立体视点不匹配失真,先对左右视点失真图分别 进行视差映射, 再提取映射图与该视点失真图的差异区域作为双目不匹配区域,然后针对不匹配区域计算MS SIM 值,最 后将左右视点平均池化作为双目竞争特征值;最终将两个特征值幂次融合,作为最终的立体 虚拟视点图像 质量评价客观指标。实验结果表明本方法有效匹配主观打分的DMOS值,皮尔森线性相关系数 和斯皮尔曼 秩相关系数分别为0.911和0.900,正确反映了 立体虚拟视点图像质量。