Quality assessment of multiply and singly distorted stereoscopic images via adaptive construction of cyclopean views

A challenging problem confronted when designing a blind/no-reference (NR) stereoscopic image quality assessment (SIQA) algorithm is to simulate the quality assessment (QA) behavior of the human visual system (HVS) during binocular vision. An effective way to solve this problem is to estimate the quality of the merged single view created in the human brain which is also referred to as the cyclopean image. However, due to the difficulty in modeling the binocular fusion and rivalry properties of the HVS, obtaining effective cyclopean images for QA is non-trivial, and consequently previous NR SIQA algorithms either require the MOS/DMOS values of the distorted 3D images for training or ignore the quality analysis of the merged cyclopean view. In this paper, we focus on (1) constructing accurate and appropriate cyclopean views for QA of stereoscopic images by adaptively analyzing the distortion information of two monocular views, and (2) training NR SIQA models without requiring the assistance of the MOS/DMOS values in existing databases. Accordingly, we present an effective opinion-unaware SIQA algorithm called MUSIQUE-3D, which blindly assesses the quality of multiply and singly distorted stereoscopic images by analyzing quality degradations of both monocular and cyclopean views. The monocular view quality is estimated by an extended version of the MUSIQUE algorithm, and the cyclopean view quality is computed from the distortion parameter values predicted by a two-layer classification-regression model trained on a large 3D image dataset. Tests on various 3D image databases demonstrate the superiority of our method as compared with other state-of-the-art SIQA algorithms.     

Stereo image quality assessment considering the difference of statistical feature in early visual pathway

Human visual theory is closely related to stereo image quality assessment (SIQA), which determines whether the evaluation results of SIQA method can keep good consistency with subjective perception. Many SIQA methods are not fully based on human visual theory, so there is still room for improvement. The research on the visual system tends to the dorsal and ventral pathways, which ignores the information differences in the early visual pathways. It is worth noting that the ON and OFF receptive fields in retinal ganglion cells (RGCs) respond asymmetrically to the statistical features of images. Inspired by this, in this paper, we propose an SIQA method based on monocular and binocular visual features, which takes into account the difference of ON and OFF response features in early visual pathways. Moreover, the different information interaction mechanisms of visual cortex are used to fuse the response maps information of left and right images. Final, monocular and binocular features are extracted and sent to support vector regression (SVR) for quality prediction. Experimental results show that the proposed method is superior to several mainstream SIQA metrics on four publicly available stereo image databases.     

No-reference stereoscopic image quality evaluator based on human visual characteristics and relative gradient orientation

Stereoscopic image quality assessment (SIQA) is of great significance to the development of modern three-dimensional (3D) display technology. In this work, by further mining the relationship between visual features and stereoscopic image quality perception, we build a new no-reference SIQA model, which combines the monocular and binocular features. Statistical quality-aware structural features from relative gradient orientation (RGO) map and texture features from the histogram of the weighted local binary pattern (LBP) in the texture image (TLBP) are not only extracted from both monocular view, but also extracted from binocular views to predict binocular quality perception. Meanwhile, the color statistical features ignored by most models and the binocularity feature is extracted to complement the monocular features and the above binocular features, respectively. Finally, all the extracted features and subjective scores are used to predict the objective quality score through the support vector regression (SVR) model. Experiments on four popular stereoscopic image databases show that the proposed model achieves high consistency with subjective assessment, and the performance of the model is very competitive with the latest models.     

Blind quality assessment for multiply distorted stereoscopic images towards IoT-based 3D capture systems

Empowered by 5G mobile communication networks, multimedia processing has been considered as a very promising application of Internet-of-Things (IoT). Stereoscopic image quality assessment (SIQA), as an important part of 3D capture system, can be embedded in the cloud or fog servers to automatically monitor the perceptual quality of the collected stereoscopic images. In this paper, a novel blind image quality assessment method towards IoT-based 3D capture systems is developed for multiply-distorted stereoscopic images (MDSIs), in which five complementary channels, including left view, right view, cyclopean map, summation map and difference map, are jointly considered in dictionary learning for characterizing the monocular receptive field (MRF) and binocular receptive field (BRF) properties of the visual cortex in response to MDSIs. Additionally, the high order statistics scheme is adopted by utilizing the statistical differences between the codebook and images to ensure the stable and robust quality prediction performance for MDSIs. The proposed method shows competitive prediction performances on four benchmark databases compared with the existing SIQA metrics.     

Two-stream interactive network based on local and global information for No-Reference Stereoscopic Image Quality Assessment

Nowadays, stereoscopic image quality assessment (SIQA) based on convolutional neural network (CNN) has become the mainstream model of image quality assessment (IQA). Compared with the two-dimensional quality evaluation model, stereoscopic image quality evaluation is more challenging due to the effects of depth and parallax information. In this paper, we propose a two-stream interactive network model to perform quality evaluation, which can well simulate the process of human stereo visual perception. Meanwhile, we enhance the extraction of local and global features of images by asymmetric convolution kernel and interactive sub-networks of inter-layers, respectively, which can further optimize our network model. Our proposed algorithm was evaluated on four public databases. The final experimental results show that our proposed algorithm exhibits good performance not only on the whole database but also on each single distortion type.     

A perceptual stereoscopic image quality assessment model accounting for binocular combination behavior

Stereoscopic image quality assessment (SIQA) plays an important role in the development of 3D image processing. In this paper, a full-reference object SIQA model is built based on binocular summation channel and binocular difference channel. In our frame work, binocular combination behavior and how to experience the depth perception are thought to be the key factors to evaluate the quality of stereoscopic images. Differing from the current depth map methods, this method focuses on a new aspect, and an effective combination model is proposed based on the physiological findings in the Human Visual System (HVS). Experimental results demonstrate that the proposed quality assessment metric significantly outperforms the existing metrics and can achieve higher consistency with subject quality assessment when predicting the quality of stereoscopic images that have been symmetrically distorted.     

基于双目特征联合的无参考立体图像质量评价

通过模拟人类视觉系统(HVS)的双目视觉行为,提 出一种基于双目特征联合的无参考立 体图像质量评价(NR-SIQA)方法。首先分析立体视觉感知中的双目联合行为,提出 可应用于立体图像质量预 测的双目联合模型;然后采用学习和统计分析的方法,分别提取局部和全局特征并联合作 为感知特征; 最后采用机器学习算法,建立特征和质量的关系模型,并结合基于特征的双目联合模型预测 立体图像质量。实验结果表明,本文方法在对称立体图像库上的Pearson线性相关系数(PLCC)和Spearman等级系数(SRCC)高于0.93,在非对称库上高于0.87,优 于现有评价方法。     

结合空间域与变换域特征提取的盲立体图像质量评价

针对立体图像质量预测准确性不足的问题,该文提出了一种结合空间域和变换域提取质量感知特征的无参考立体图像质量评价模型。在空间域和变换域分别提取输入的左、右视图的自然场景统计特征,并在变换域提取合成独眼图的自然场景统计特征,然后将其输入到支持向量回归(SVR)中,训练从特征域到质量分数域的预测模型,并以此建立SIQA客观质量评价模型。在4个公开的立体图像数据库上与一些主流的立体图像质量评价算法进行对比,以在LIVE 3D Phase I图像库中的性能测试为例,Spearman秩相关系数、皮尔逊线性相关系数和均方根误差分别达到0.967,0.946和5.603,验证了所提算法的有效性。     

No-reference stereoscopic images quality assessment method based on monocular superpixel visual features and binocular visual features

No-reference quality assessment of images has received considerable attention. However, the accuracy of such assessment remains questionable because of its weak biological basis. In this paper, we propose a novel quality assessment model based on the superpixel index and biological binocular mechanisms. The technical contributions of our model are the introduction of local monocular superpixel features and three global binocular visual features. We utilize monocular superpixel segmentation to extract two types of entropies as the local visual features for accurate quality-aware feature extraction. In addition, natural scene statistics features are extracted from the binocular visual information to complement the local monocular features and quantify the naturalness of the stereoscopic images. Finally, a regression model is learned to evaluate the quality of the stereoscopic images. Experimental results from three popular databases demonstrate that the proposed model has a more reliable performance than earlier models in terms of prediction accuracy and generalizability.     

基于深层特征学习的无参考立体图像质量评价

提出了一种基于深层特征学习的无参考(NR)立体图 像质量评价方 法。与传统人工提取图像特征不同,采用卷积神经网络(CNN)自动提取图像特征,评价过程 分为训练和 测试两阶段。在训练阶段,将图像分块训练CNN网络,利用CNN提取图像块特征,并结合不同 的整合方式 得到图像的全局特征,通过支持向量回归(SVR)建立主观质量与全局特征的回归模型;在测 试阶段,由已训练的CNN网 络和回归模型,得到左右图像和独眼图的质量。最后,根据人眼双目视觉特性融合左图像、 右图像和独眼 图的质量,得到立体图像质量。本文方法在LIVE-I和LIVE-II数据库上的Spearman等级系 数(SROCC)分别达 到了0.94,评价结果准确,与人眼的主 观感受一致。     

Deep network based stereoscopic image quality assessment via binocular summing and differencing

With the development of deep networks in dealing with various visual tasks, the deep network based on binocular vision is expected to tackle the issue of stereoscopic image quality assessment. Here, we present a stereoscopic image quality assessment method using the deep network with four channels together, which takes the left view, right view, binocular summing view, and binocular differencing view as the inputs of the network. The visual features are enhanced through the concatenation in a weighted way, so that the binocular vision can be adequately included in the binocular addition and subtraction information. Compared with the state-of-the-art metrics, the proposed method exhibits relatively high performances on four benchmark databases.     

基于三维感知的立体虚拟视点图像质量评价方法

为了评价立体虚拟视点图像的质量,提出了一种基 于三维感知的客观评价方法。综合考虑了立体虚拟视点图像两大最主要失真类型:单视点绘 制失真和立体视点不匹配失真。针对单视点绘制失真,先提取 当前视点失真图与无失真图的差异性区域,再针对该差异性区域计算平均结构相似度(MSSIM ),最后将左 右视点平均池化作为单目纹理特征值;针对立体视点不匹配失真,先对左右视点失真图分别 进行视差映射, 再提取映射图与该视点失真图的差异区域作为双目不匹配区域,然后针对不匹配区域计算MS SIM 值,最 后将左右视点平均池化作为双目竞争特征值;最终将两个特征值幂次融合,作为最终的立体 虚拟视点图像 质量评价客观指标。实验结果表明本方法有效匹配主观打分的DMOS值,皮尔森线性相关系数 和斯皮尔曼 秩相关系数分别为0.911和0.900,正确反映了 立体虚拟视点图像质量。     

基于双目能量模型的立体图像质量客观评价方法

立体图像的深度感知取决于双目差距,基础视觉皮层的差距调谐细胞在感知立体深度的过程中起着关键作用.单只眼睛感知的图像信息被单目简单细胞接收之后传入双目简单细胞进行处理,继而被送入复杂细胞进行综合,得到基础视觉皮层对于一幅立体图像的能量响应.用数学模型来仿真简单的以及复杂的细胞对于双目差距的响应,并通过双目能量模型的计算达到评价彩色立体图像的目的.实验结果表明,双目能量模型的评价结果与立体图像的主观评价值具有较高的一致性.     

基于稀疏表示的立体图像客观质量评价方法

提出了一种基于稀疏表示的立体图像质量评价方法 ,分为训练和测试两个部分。在训练部 分,通过训练不同频带的立体图像获得立体图像的稀疏字典;在测试部分,根据稀疏字典计 算得到立体图 像的稀疏特征,定义了稀疏特征相似度衡量原始和失真图像信息的差异,并根据稀疏字典计 算了频带增益和左右视点的融合权值,最后融合稀疏特征相似度作为立体图像质量的 客观评价值。在立体图像测试库上的实验结果表明,本文方法的评价结果与主观评价结果有 较好的相关性,符合人类视觉系统的感知。     

No-reference Stereoscopic Image Quality Assessment Using Binocular Self-similarity and Deep Neural Network

Quality assessment of three-dimensional (3D) images is more challenging than that of 2D images. The quality of 3D visual experience is one of the most challenging areas of human binocular perception and is affected by multiple factors such as asymmetric stereo image/video compression, depth perception, visual discomfort, and single view quality. In this paper, we propose a new no-reference quality assessment method for stereoscopic images based on Binocular Self-similarity (BS) and Deep Neural Networks (DNN). To be more specific, a BS index is defined and computed according to binocular rivalry and suppression based on the depth image-based rendering technique. Then, a DNN is trained in an opinion unaware way to predict local quality. Binocular integration (BI) index is calculated by using the trained DNN, accounting for binocular integration behaviors. Finally, the final quality score of stereoscopic image is obtained by combining the BS and BI indexes together. Experimental results on four public 3D image quality assessment databases demonstrate that compared with existing methods, the proposed method can achieve high consistency with subjective perception on stereoscopic images with both symmetric and asymmetric distortions.     

基于未匹配子带合成系数统计特性的立体图像质量评价算法

针对经典立体图像质量评价算法存在评价准确性 低以及特征提取耗时较长的问题, 提出一种基于未匹配子带合成系数统计特性的立体图像质量评价算法。首先,利用可控金字 塔对左右视点图像进行多尺度、多方向的小波分解,并将左右视点图像在相同尺度、相同方 向上未经视差图匹配的小波子带系数合成为子带合成系数。其次,提取小波子带合成系数中 的统计分布特征,相同尺度相邻方向小波子带合成系数之间的相关性特征,以及相同方向相 邻尺度子带合成系数之间的相关性特征。最后,利用所提取特征训练经典的支持向量回归模 型,预测图像质量。在LIVE 3D和Waterloo IVC 3D数据库上的实验结果表明,与主流立体 图像质量评价算法相比,本文算法在预测对称和非对称失真立体图像质量时都获得了更高的 评价准确性。同时,由于子带合成系数的生成无需根据视差图进行匹配,算法执行效率高。     

No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks

No-reference image quality assessment is of great importance to numerous image processing applications, and various methods have been widely studied with promising results. These methods exploit handcrafted features in the transformation or space domain that are discriminated for image degradations. However, abundant a priori knowledge is required to extract these handcrafted features. The convolutional neural network (CNN) is recently introduced into the no-reference image quality assessment, which integrates feature learning and regression into one optimization process. Therefore, the network structure generates an effective model for estimating image quality. However, the image quality score obtained by the CNN is based on the mean of all of the image patch scores without considering the human visual system, such as edges and contour of images. In this paper, we combine the CNN and the Prewitt magnitude of segmented images and obtain the image quality score using the mean of all the products of the image patch scores and weights based on the result of segmented images. Experimental results on various image distortion types demonstrate that the proposed algorithm achieves good performance.     

基于双目能量响应的无参考立体图像质量评价

为了实现对不同失真类型立体图像的质量评价,提出了一种基于双目能量响应的无参考立体图像质量评价(NR-IAQ)方法。首先,通过对各失真图像进行Gabor滤波,提取出不同频率、不同方向、不同视差响应下的局部特征矢量,作为立体图像特征信息;然后,利用支持向量回归(SVR)建立立体图像特征与主观评价值的关系,从而预测得到立体图像质量的客观评价值。实验结果表明,对于NBU-3D测试库,Pearson线性相关系数值在0.92以上,Spearman等级相关系数值在0.93以上;对于LIVE-3D测试库,Pearson线性相关系数值在0.96以上,Spearman等级相关系数值在0.96以上;与现有的全参考(FR)和(NR)质量评价方法相比,本方法得到的客观评价值与主观评价结果有较好的相关性,更加符合人眼视觉系统。     

Blind S3D image quality prediction using classical and non-classical receptive field models

We develop a novel no-reference image quality assessment model for stereoscopic 3D (S3D) images that is inspired by functional receptive field models of perceptual mechanisms in primary visual cortex (V1). The approach is called the Blind S3D Integrated Quality Evaluator (BSIQE). BSIQE simulates monocular and binocular responses to stereo views using channel separation and weighted multi-channel combination models. Binocular responses are modeled as the fusion of the two channels using a weighted multi-channel combination. The responses to stereoscopic image content of both classical and non-classical anisotropic receptive fields are then modeled based on a determination of the relative importance of the receptive field responses. In the last stage of feature extraction, we deploy a simple and efficient way of decorrelating the picture data. We extract local binary pattern (LBP) statistical features from the computed receptive field responses, and use them to train a regressor to predict the perceptual quality of stereoscopic images. We carefully evaluate BSIQE on four public-domain 3D image quality databases, and find that it is statistically superior to all compared 2D and 3D IQA algorithms. BSIQE exhibits good performance across the datasets suggesting that it is general, and it has relatively low complexity.     

Depth estimation of supervised monocular images based on semantic segmentation

In recent years, the research method of depth estimation of target images using Convolutional Neural Networks (CNN) has been widely recognized in the fields of artificial intelligence, scene understanding and three-dimensional (3D) reconstruction. The fusion of semantic segmentation information and depth estimation will further improve the quality of acquired depth images. However, how to deeply combine image semantic information with image depth information and use image edge information more accurately to improve the accuracy of depth image is still an urgent problem to be solved. For this purpose, we propose a novel depth estimation model based on semantic segmentation to estimate the depth of monocular images in this paper. Firstly, a shared parameter model of semantic segmentation information and depth estimation information is built, and the semantic segmentation information is used to guide depth acquisition in an auxiliary way. Then, through the multi-scale feature fusion module, the feature information contained in the neural network on different layers is fused, and the local feature information and global feature information are effectively used to generate high-resolution feature maps, so as to achieve the goal of improving the quality of depth image by optimizing the semantic segmentation model. The experimental results show that the model can fully extract and combine the image feature information, which improves the quality of monocular depth vision estimation. Compared with other advanced models, our model has certain advantages.