基于深层特征学习的无参考立体图像质量评价

提出了一种基于深层特征学习的无参考(NR)立体图 像质量评价方 法。与传统人工提取图像特征不同,采用卷积神经网络(CNN)自动提取图像特征,评价过程 分为训练和 测试两阶段。在训练阶段,将图像分块训练CNN网络,利用CNN提取图像块特征,并结合不同 的整合方式 得到图像的全局特征,通过支持向量回归(SVR)建立主观质量与全局特征的回归模型;在测 试阶段,由已训练的CNN网 络和回归模型,得到左右图像和独眼图的质量。最后,根据人眼双目视觉特性融合左图像、 右图像和独眼 图的质量,得到立体图像质量。本文方法在LIVE-I和LIVE-II数据库上的Spearman等级系 数(SROCC)分别达 到了0.94,评价结果准确,与人眼的主 观感受一致。     

一种基于最小可察觉失真的立体图像质量客观评价方法

立体图像质量是评价立体视频系统性能的有效途径,而如何利用人类视觉特性对立体图像质量进行有效的评价是目前的研究难点。本文通过分析最小可察觉失真(JND,just noticeable distortion)视觉感知模型,并结合反映图像结构信息的奇异值矢量,提出了一种基于JND的立体图像质量客观评价方法。评价方法由图像质量评价和深度感知评价两部分组成,首先提取反映图像质量和深度感知的特征信息作为立体图像特征信息,然后根据立体图像的不同失真类型情况对其特征进行融合,通过支持向量回归(SVR,support vector Regression)预测得出立体图像质量的客观评价值。实验结果表明,采用本文提出的客观评价方法对立体数据测试库进行评价,在不同失真类型或混合失真评价结果中,Pearson线性相关系数(CC)值均在0.94以上,Spearman等级相关系数(SROCC)值均在0.92以上,符合人眼视觉特性,能够很好地预测人眼对立体图像的主观感知。     

Robust image retrieval by cascading a deep quality assessment network

The performance of computer vision algorithms can severely degrade in the presence of a variety of distortions. While image enhancement algorithms have evolved to optimize image quality as measured according to human visual perception, their relevance in maximizing the success of computer vision algorithms operating on the enhanced image has been much less investigated. We consider the problem of image enhancement to combat Gaussian noise and low resolution with respect to the specific application of image retrieval from a dataset. We define the notion of image quality as determined by the success of image retrieval and design a deep convolutional neural network (CNN) to predict this quality. This network is then cascaded with a deep CNN designed for image denoising or super resolution, allowing for optimization of the enhancement CNN to maximize retrieval performance. This framework allows us to couple enhancement to the retrieval problem. We also consider the problem of adapting image features for robust retrieval performance in the presence of distortions. We show through experiments on distorted images of the Oxford and Paris buildings datasets that our algorithms yield improved mean average precision when compared to using enhancement methods that are oblivious to the task of image retrieval. ¹     

No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks

No-reference image quality assessment is of great importance to numerous image processing applications, and various methods have been widely studied with promising results. These methods exploit handcrafted features in the transformation or space domain that are discriminated for image degradations. However, abundant a priori knowledge is required to extract these handcrafted features. The convolutional neural network (CNN) is recently introduced into the no-reference image quality assessment, which integrates feature learning and regression into one optimization process. Therefore, the network structure generates an effective model for estimating image quality. However, the image quality score obtained by the CNN is based on the mean of all of the image patch scores without considering the human visual system, such as edges and contour of images. In this paper, we combine the CNN and the Prewitt magnitude of segmented images and obtain the image quality score using the mean of all the products of the image patch scores and weights based on the result of segmented images. Experimental results on various image distortion types demonstrate that the proposed algorithm achieves good performance.     

基于支持向量回归的立体图像客观质量评价模型

立体图像质量评价是评价立体视频系统性能的有效途径,而如何利用人类视觉特性对立体图像质量进行有效评价是目前的研究难点。该文根据图像奇异值有较强稳定性的特点,结合立体图像的主观视觉特性,提出了一种基于支持向量回归(Support Vector Regression, SVR)的立体图像客观质量评价模型。该模型通过分析立体图像的视觉特性,提取左右图像的奇异值作为立体图像的特征信息,然后根据立体图像的不同失真类型情况对其特征进行融合,通过SVR预测得到立体图像质量的客观评价值。实验结果表明,采用该文提出的客观评价模型对立体数据测试库进行评价,Pearson线性相关系数值在0.93以上,Spearman等级相关系数值在0.94以上,均方根误差值接近6,异常值比率值为0.00%,符合人眼视觉特性,能够很好地预测人眼对立体图像的主观感知。     

A fuzzy convolutional neural network for enhancing multi-focus image fusion

The images captured by the cameras contain distortions, misclassified pixels, uncertainties and poor contrast. Therefore, the multi-focus image fusion (MFIF) integrates various input image features to produce a single fused image using all its objects in focus. However, it is computationally complex, which leads to inconsistency. Hence, the MFIF method is employed to generate the fused image by integrating the fuzzy sets (FS) and convolutional neural network (CNN) to detect focused and unfocused parts in both source images. It is also compared with other competing six MFIF methods like Neutrosophic set based stationary wavelet transform (NSWT), guided filters, CNN, ensemble CNN, image fusion-based CNN and deep regression pair learning (DRPL). Benchmark datasets validate the superiority of the proposed FCNN method in terms of four non-reference assessment measures having mutual information (1.1678), edge information (0.7281), structural similarity (0.9850) and human perception (0.8020) and two reference metrics such as Peak signal-to-noise ratio (57.23) and root mean square error (1.814).     

基于双眼视觉感知特征一致性的立体图像体验质量评价

3D图像被认为是多媒体技术的重要标志,其中,立体图像质量对3D图像发展起到至关重要的作用。不同于传统的2D图像质量评价,在3D图像质量评价中引入关于体验质量（ QoE）问题的新挑战,因此,本文提出一个基于双眼视觉感知特征一致性的立体图像体验质量评价算法。具体地,先对2个视点图像提取像素梯度作为视觉感知的低层次特征,再用梯度方向直方图特征（ HOG）建立立体图像的视觉感知特征向量,然后,由支持向量回归（ SVR）方法来学习视觉感知特征与立体图像体验质量得分的关系,最后,通过训练好的SVR模型来预测立体图像体验质量。实验结果表明所提算法能够有效地预测立体图像体验质量。     

基于支持向量回归的图像超分辨率重建算法

文章提出一种新的基于支持向量回归(SVR)和稀疏表示的图像超分辨重建算法。SVR对输入数据有良好预测输出类别能力。图像统计表明,图像块可以从过完备字典中通过稀疏线性组合很好的表示。对一幅低分辨率输入图像,可以将图像超分辨问题视为在高分辨图像中估计其像素位置。与传统的支持向量回归方法相比,本文采用的特征是不同类型的图像块的稀疏表示。研究表明,稀疏表示作为特征对噪声有一定的鲁棒性。实验结果表明,本文方法与传统支持向量回归方法相比在图像重建质量上有一定的优势。     

基于视觉感知和零水印的部分参考立体图像质量评价模型

通过分析人类视觉系统的纹理方向特性和立体感知特性,并结合数字水印的半脆弱性和支持向量回归(Support Vector Regression, SVR)的泛化学习能力,该文提出一种基于视觉感知和零水印的部分参考立体图像质量客观评价模型。该模型利用立体图像左右视点经小波分解后在同一空间频率的水平和垂直方向子带系数关系构造反映图像纹理方向特征的视点零水印,同时,利用左右视点视差值与自适应阈值的大小关系构造反映立体感质量的视差零水印,然后利用SVR来学习两类零水印恢复率(视觉加权视点零水印恢复率和视差零水印恢复率)与主观评价值的关系,最后用训练好的SVR完成立体图像质量预测。实验结果表明该模型符合人眼视觉特性,所得到的客观评价值与主观评价值具有较好的一致性。     

基于视差空间图的立体图像质量客观评价方法

立体图像质量评价是评价立体视频系统性能的有 效途径,而如何利用人类视觉特性对立体图像质量 进行有效评价是目前的研究难点。本文提出了一种基于视差空间图(DSI) 的立体图像质量客观评价方法。首先, 分别构造原始立体图像和失真立体图像的DSI图;然后,通过三维离散余弦变换(3D-DCT)提取出反映图像质量 和深度感知的特征信息,并采用主成分分析(PCA)进行特征降维,形成立体图像特征信息; 最后,通过支持向量 回归(SVR)建立立体图像特征与主观评价值的关系,从而预测得到立体图像质量的客观评价 值。实验表明, 对于对称立体图像库,Pearson线性相关系数(PLCC)和Spe arman等级相关系数(SROCC)值均达到0.94以上;对于非 对称立体图像库,PLCC和SROCC值分别达到0.94以上。结果表明,本文方法能够很好地预测人眼对立体图像的主观感 知。     

Two-stream interactive network based on local and global information for No-Reference Stereoscopic Image Quality Assessment

Nowadays, stereoscopic image quality assessment (SIQA) based on convolutional neural network (CNN) has become the mainstream model of image quality assessment (IQA). Compared with the two-dimensional quality evaluation model, stereoscopic image quality evaluation is more challenging due to the effects of depth and parallax information. In this paper, we propose a two-stream interactive network model to perform quality evaluation, which can well simulate the process of human stereo visual perception. Meanwhile, we enhance the extraction of local and global features of images by asymmetric convolution kernel and interactive sub-networks of inter-layers, respectively, which can further optimize our network model. Our proposed algorithm was evaluated on four public databases. The final experimental results show that our proposed algorithm exhibits good performance not only on the whole database but also on each single distortion type.     

No-reference stereoscopic image quality evaluator based on human visual characteristics and relative gradient orientation

Stereoscopic image quality assessment (SIQA) is of great significance to the development of modern three-dimensional (3D) display technology. In this work, by further mining the relationship between visual features and stereoscopic image quality perception, we build a new no-reference SIQA model, which combines the monocular and binocular features. Statistical quality-aware structural features from relative gradient orientation (RGO) map and texture features from the histogram of the weighted local binary pattern (LBP) in the texture image (TLBP) are not only extracted from both monocular view, but also extracted from binocular views to predict binocular quality perception. Meanwhile, the color statistical features ignored by most models and the binocularity feature is extracted to complement the monocular features and the above binocular features, respectively. Finally, all the extracted features and subjective scores are used to predict the objective quality score through the support vector regression (SVR) model. Experiments on four popular stereoscopic image databases show that the proposed model achieves high consistency with subjective assessment, and the performance of the model is very competitive with the latest models.     

结合空间域与变换域特征提取的盲立体图像质量评价

针对立体图像质量预测准确性不足的问题,该文提出了一种结合空间域和变换域提取质量感知特征的无参考立体图像质量评价模型。在空间域和变换域分别提取输入的左、右视图的自然场景统计特征,并在变换域提取合成独眼图的自然场景统计特征,然后将其输入到支持向量回归(SVR)中,训练从特征域到质量分数域的预测模型,并以此建立SIQA客观质量评价模型。在4个公开的立体图像数据库上与一些主流的立体图像质量评价算法进行对比,以在LIVE 3D Phase I图像库中的性能测试为例,Spearman秩相关系数、皮尔逊线性相关系数和均方根误差分别达到0.967,0.946和5.603,验证了所提算法的有效性。     

Quality assessment for color correction-based stitched images via bi-directional matching

With the deepening of social information, the panoramic image has drawn a significant interest of viewers and researchers as it can provide a very wide field of view (FoV). Since panoramic images are usually obtained by capturing images with the overlapping regions and then stitching them together, image stitching plays an important role in generating panoramic images. In order to effectively evaluate the quality of stitched images, a novel quality assessment method based on bi-directional matching is proposed for stitched images. Specifically, dense correspondences between the testing and benchmark stitched images are first established by bi-directional SIFT-flow matching. Then, color-aware, geometric-aware and structure-aware features are respectively extracted and fused via support vector regression (SVR) to obtain the final quality score. Experiments on our newly constructed database and ISIQA database demonstrate that the proposed method can achieve comparable performance compared with the conventional blind quality metrics and the quality metrics specially designed for stitched images.     

Depth estimation of supervised monocular images based on semantic segmentation

In recent years, the research method of depth estimation of target images using Convolutional Neural Networks (CNN) has been widely recognized in the fields of artificial intelligence, scene understanding and three-dimensional (3D) reconstruction. The fusion of semantic segmentation information and depth estimation will further improve the quality of acquired depth images. However, how to deeply combine image semantic information with image depth information and use image edge information more accurately to improve the accuracy of depth image is still an urgent problem to be solved. For this purpose, we propose a novel depth estimation model based on semantic segmentation to estimate the depth of monocular images in this paper. Firstly, a shared parameter model of semantic segmentation information and depth estimation information is built, and the semantic segmentation information is used to guide depth acquisition in an auxiliary way. Then, through the multi-scale feature fusion module, the feature information contained in the neural network on different layers is fused, and the local feature information and global feature information are effectively used to generate high-resolution feature maps, so as to achieve the goal of improving the quality of depth image by optimizing the semantic segmentation model. The experimental results show that the model can fully extract and combine the image feature information, which improves the quality of monocular depth vision estimation. Compared with other advanced models, our model has certain advantages.     

Stereo image quality assessment considering the difference of statistical feature in early visual pathway

Human visual theory is closely related to stereo image quality assessment (SIQA), which determines whether the evaluation results of SIQA method can keep good consistency with subjective perception. Many SIQA methods are not fully based on human visual theory, so there is still room for improvement. The research on the visual system tends to the dorsal and ventral pathways, which ignores the information differences in the early visual pathways. It is worth noting that the ON and OFF receptive fields in retinal ganglion cells (RGCs) respond asymmetrically to the statistical features of images. Inspired by this, in this paper, we propose an SIQA method based on monocular and binocular visual features, which takes into account the difference of ON and OFF response features in early visual pathways. Moreover, the different information interaction mechanisms of visual cortex are used to fuse the response maps information of left and right images. Final, monocular and binocular features are extracted and sent to support vector regression (SVR) for quality prediction. Experimental results show that the proposed method is superior to several mainstream SIQA metrics on four publicly available stereo image databases.     

On the use of a scanpath predictor and convolutional neural network for blind image quality assessment

Image quality assessment is an important field in computer vision, since it has a great impact on related tasks. To meet these needs, a plethora of metrics has been developed. In this paper, we propose an efficient method that estimates the quality of 2D images without access to the pristine image. This metric is modeled based on the relevant patches selected by saliency information and a convolution neural network. To exploit the saliency information, only the more perceptually relevant patches that impact subjective judgment more, are considered. To this end, we first compute the saliency map of the distorted image. Then, a scanpath predictor that aims to mimic the visual behavior is employed as patch selector. Finally, a CNN model is used to predict the quality score through the extracted patches. To the best of our knowledge this is the first study to associate a scanpath prediction method and CNN to assess the quality of 2D images. Four CNN models were compared (AlexNet, VGG16, VGG19 and ResNet50) and the performance of the best CNN was compared to the state-of-the-art on four datasets. Experimental results demonstrated the efficiency of the proposed approach and its generalization capacity.     

A no-reference perceptual image quality assessment database for learned image codecs

The drastic growth of research in image compression, especially deep learning-based image compression techniques, poses new challenges to objective image quality assessment (IQA). Typical artifacts encountered in the emerging image codecs are significantly different from that produced by traditional block-based codecs, leading to inapplicability of the existing objective IQA algorithms. Towards advancing the development of objective IQA algorithms for recent compression artifacts, we built a learning-based compressed image quality assessment (LCIQA) database involving traditional block-based image codecs, hybrid neural network based image codecs, convolutional neural network based and generative adversarial network (GAN) based end-to-end optimized image coding approaches. Our study confirms the statistical difference and human perception difference between reconstructions of learned compression and traditional block-based compression. We propose a two-step deep learning model for learning-based compressed image quality assessment. Extensive experiments on LCIQA database demonstrate that our proposed model performs better than other counterparts on learning-based compressed images, especially on GAN compressed images, and achieves competitive performance to the state-of-the-art IQA metrics on traditional compressed images.