No-reference stereoscopic images quality assessment method based on monocular superpixel visual features and binocular visual features

No-reference quality assessment of images has received considerable attention. However, the accuracy of such assessment remains questionable because of its weak biological basis. In this paper, we propose a novel quality assessment model based on the superpixel index and biological binocular mechanisms. The technical contributions of our model are the introduction of local monocular superpixel features and three global binocular visual features. We utilize monocular superpixel segmentation to extract two types of entropies as the local visual features for accurate quality-aware feature extraction. In addition, natural scene statistics features are extracted from the binocular visual information to complement the local monocular features and quantify the naturalness of the stereoscopic images. Finally, a regression model is learned to evaluate the quality of the stereoscopic images. Experimental results from three popular databases demonstrate that the proposed model has a more reliable performance than earlier models in terms of prediction accuracy and generalizability.     

Objective quality assessment of synthesized images by local variation measurement

Due to the rapid development of free-viewpoint television (FVT), Depth-Image-Based Rendering (DIBR) technology has been widely used to synthesize images of virtual view-points. However, the types of distortions in the synthesized images are different from those of natural images, such as discontinuity, flickering, stretching, etc. To measure the distortion occurred in the synthesized images, we propose a full-reference (FR) quality assessment method by local variation measurement consisting of three-modules. Firstly, since the distortion in the synthesized image mainly occurs in the region with high-frequency structure information, the Neutrosophic domain is employed to evaluate the degradation of local image structure. Secondly, by considering that the texture of the synthesized image might be damaged due to the warping of 2D image or the loss of information in the occlusion region, we evaluate the visual quality of local texture by using the features obtained from frequency domain. Thirdly, to measure the stretching distortion which is unique in the synthesized image, the visual quality of extracted stretching area is measured by entropy. Finally, a pooling operation is used to combine the quality scores of the three modules to obtain the final predicted quality score. Experimental results show that the performance of the proposed algorithm is competitive with state-of-the-art FR and no-reference image quality assessment metrics.     

Geometry-based distributed scene representation with omnidirectional vision sensors

This paper addresses the problem of efficient representation of scenes captured by distributed omnidirectional vision sensors. We propose a novel geometric model to describe the correlation between different views of a 3-D scene. We first approximate the camera images by sparse expansions over a dictionary of geometric atoms. Since the most important visual features are likely to be equivalently dominant in images from multiple cameras, we model the correlation between corresponding features in different views by local geometric transforms. For the particular case of omnidirectional images, we define the multiview transforms between corresponding features based on shape and epipolar geometry constraints. We apply this geometric framework in the design of a distributed coding scheme with side information, which builds an efficient representation of the scene without communication between cameras. The Wyner-Ziv encoder partitions the dictionary into cosets of dissimilar atoms with respect to shape and position in the image. The joint decoder then determines pairwise correspondences between atoms in the reference image and atoms in the cosets of the Wyner-Ziv image in order to identify the most likely atoms to decode under epipolar geometry constraints. Experiments demonstrate that the proposed method leads to reliable estimation of the geometric transforms between views. In particular, the distributed coding scheme offers similar rate-distortion performance as joint encoding at low bit rate and outperforms methods based on independent decoding of the different images.     

注意力模型指导下的视频质量评价方法

在视频质量评价方法中,常常需要对人眼视觉系统做出合理的假设,其中注意力模型就是一个很重要的因素。提出了一种在注意力模型指导下的视频质量评价方法,在图像帧的质量评价中加入了显著性区域信息,使之更能符合人眼视觉特性,并兼顾了视频中的运动信息,在一定程度上提高了客观质量评价方法的性能。     

BGT: A blind image quality evaluator via gradient and texture statistical features

Blind image quality assessment (BIQA) aims to design a model that can accurately evaluate the quality of the distorted image without any information about its reference image. Previous studies have shown that gradients and textures of image is widely used in image quality evaluation tasks. However, few studies used the joint statistics of gradient and texture information to evaluate image quality. Considering the visual perception characteristics of the human visual system, we develop a novel general-purpose BIQA model via two sets of complementary perception features. Specifically, the joint statistical histograms of gradient and texture are extracted as the first set of features, and the second set of features is extracted using the local binary pattern (LBP) operator. After extracting two groups of complementary quality-aware features, the feature vectors are sent to the support vector regression machine to establish the nonlinear relationship between quality-aware features and quality scores. A large number of experiments on seven large benchmark databases show that the proposed BIQA model has higher accuracy, better generalization properties and lower computational complexity than the relevant state-of-the-art BIQA metrics.     

基于支持向量回归的立体图像客观质量评价模型

立体图像质量评价是评价立体视频系统性能的有效途径,而如何利用人类视觉特性对立体图像质量进行有效评价是目前的研究难点。该文根据图像奇异值有较强稳定性的特点,结合立体图像的主观视觉特性,提出了一种基于支持向量回归(Support Vector Regression, SVR)的立体图像客观质量评价模型。该模型通过分析立体图像的视觉特性,提取左右图像的奇异值作为立体图像的特征信息,然后根据立体图像的不同失真类型情况对其特征进行融合,通过SVR预测得到立体图像质量的客观评价值。实验结果表明,采用该文提出的客观评价模型对立体数据测试库进行评价,Pearson线性相关系数值在0.93以上,Spearman等级相关系数值在0.94以上,均方根误差值接近6,异常值比率值为0.00%,符合人眼视觉特性,能够很好地预测人眼对立体图像的主观感知。     

Unsupervised blind image quality assessment based on joint structure and natural scene statistics features

Compared with the widely used supervised blind image quality assessment (BIQA) models, unsupervised BIQA models require little prior knowledge for calculating the objective quality scores of distorted images. In this paper, we propose an unsupervised BIQA method that aims to achieve both good performance and generalization capability with low computational complexity. Carefully selected and extensive structure and natural scene statistics (NSS) features can better represent image quality. First, we employ phase congruency (PC) and finely selected gradient magnitude map and Laplacian of Gaussian response (GM-LOG) features to represent image structure information. Second, we calculate the local mean-subtracted and contrast-normalized (MSCN) coefficients and the Karhunen–Loéve transform (KLT) coefficients to represent the naturalness of the distorted images. Last, multivariate Gaussian (MVG) model with joint features extracted from both the pristine images and the distorted images is adopted to calculate the objective image quality. Extensive experiments conducted on nine IQA databases demonstrate that the proposed method achieves better performance than the state-of-the-art BIQA methods.     

Two-stream interactive network based on local and global information for No-Reference Stereoscopic Image Quality Assessment

Nowadays, stereoscopic image quality assessment (SIQA) based on convolutional neural network (CNN) has become the mainstream model of image quality assessment (IQA). Compared with the two-dimensional quality evaluation model, stereoscopic image quality evaluation is more challenging due to the effects of depth and parallax information. In this paper, we propose a two-stream interactive network model to perform quality evaluation, which can well simulate the process of human stereo visual perception. Meanwhile, we enhance the extraction of local and global features of images by asymmetric convolution kernel and interactive sub-networks of inter-layers, respectively, which can further optimize our network model. Our proposed algorithm was evaluated on four public databases. The final experimental results show that our proposed algorithm exhibits good performance not only on the whole database but also on each single distortion type.     

Image information and visual quality.

Measurement of visual quality is of fundamental importance to numerous image and video processing applications. The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. Image QA algorithms generally interpret image quality as fidelity or similarity with a "reference" or "perfect" image in some perceptual space. Such "full-reference" QA methods attempt to achieve consistency in quality prediction by modeling salient physiological and psychovisual features of the human visual system (HVS), or by signal fidelity measures. In this paper, we approach the image QA problem as an information fidelity problem. Specifically, we propose to quantify the loss of image information to the distortion process and explore the relationship between image information and visual quality. QA systems are invariably involved with judging the visual quality of "natural" images and videos that are meant for "human consumption." Researchers have developed sophisticated models to capture the statistics of such natural signals. Using these models, we previously presented an information fidelity criterion for image QA that related image quality with the amount of information shared between a reference and a distorted image. In this paper, we propose an image information measure that quantifies the information that is present in the reference image and how much of this reference information can be extracted from the distorted image. Combining these two quantities, we propose a visual information fidelity measure for image QA. We validate the performance of our algorithm with an extensive subjective study involving 779 images and show that our method outperforms recent state-of-the-art image QA algorithms by a sizeable margin in our simulations. The code and the data from the subjective study are available at the LIVE website.     

Blind tone mapped image quality assessment with image segmentation and visual perception

With tone mapping, high dynamic range (HDR) image contents can be displayed on low dynamic range (LDR) display devices, in which some important visual information may be distorted. Thus, the tone mapped image (TMI) quality assessment is one of important issues in HDR image/video processing fields. Considering the difference of visual distortion degrees between the flat and complex regions in TMI, and considering that high-quality TMI should preserve as much information as possible of its original HDR image especially in the high/low luminance regions, this paper proposes a new blind TMI quality assessment method with image segmentation and visual perception. First, we design different features to describe the distortion of TMI’s different regions with two kinds of TMI segmentation. Then, considering that there lacks an efficient algorithm to quantify the importance of features, a feature clustering scheme is designed to eliminate the poor effect feature components in the extracted features to improve the effectiveness of the selected features. Finally, considering the diversity of tone mapping operator (TMO), which may cause global and local distortion of TMI, some other global features are also combined. At last, a final feature vector is formed to synthetically describe the distortion in TMI and used to blindly predict the TMI’s quality. Experimental results in the public ESPL-LIVE HDR database show that the Pearson linear correlation coefficient and Spearman rank order correlation coefficient of the proposed method reach 0.8302 and 0.7887, respectively, which is superior to the state-of-the-art blind TMI quality assessment methods, and it means that the proposed method is highly consistent with human visual perception.     

基于全局与局部感知特征的三维彩色点云质量评价

作为物理对象在三维空间的有效表示方法,三维彩色点云可以提供丰富的沉浸式视觉体验,但在其获取、处理、编码传输等各环节会引入失真,从而导致其视觉质量下降.因此,如何监测彩色点云的视觉质量是一个亟待解决的重要问题.本文将三维彩色点云投影到二维平面,提出了一种基于全局与局部感知特征的彩色点云视觉质量评价方法.首先,将三维彩色点云转化为彩色纹理投影图与几何投影图.然后,根据三维彩色点云的纹理与几何失真在其投影图中的不同表象,分别描述并提取其失真特征;其中,在彩色纹理投影图中提取全局颜色与局部纹理特征,在几何投影图中提取全局与局部几何特征.最后,将所有全局和局部感知特征构成最终的特征向量预测彩色点云的视觉质量.在两个主观评价数据库(SJTU-PCQA、CPCD2.0)进行测试的实验结果表明,所提出方法在性能上优于13个现有代表性视觉质量评价方法,与主观感知质量有更好的一致性.     

M2OVQA: Multi-space signal characterization and multi-channel information aggregation for quality assessment of compressed omnidirectional videos

Considering the high requirements for omnidirectional video compression, we propose an objective quality evaluation method to assess quality loss in encoding omnidirectional videos. According to characteristics of 360° videos, we consider multi-space signal characterization (MSSC) to fully characterize the distortions of video signals from spatial/image domains to frequency domains and from image content to motion information, and further consider multi-channel information aggregation (MCIA) to fuse scores from multiple projection planes and temporal divided groups. The main innovation of our method is to establish a universal framework in bridging the connection between typical quality assessment and 360° quality assessment to measure 360° video quality effectively and efficiently. Experimental results show that our method outperforms state-of-the-art 2D quality metrics and quality metrics for omnidirectional images.     

MLSIM: A Multi-Level Similarity index for image quality assessment

Image quality assessment (IQA) is of great importance to numerous image processing applications, and various methods have been proposed for it. In this paper, a Multi-Level Similarity (MLSIM) index for full reference IQA is proposed. The proposed metric is based on the fact that human visual system (HVS) distinguishes the quality of an image mainly according to the details given by low-level gradient information. In the proposed metric, the Prewitt operator is first utilized to get gradient information of both reference and distorted images, then the gradient information of reference image is segmented into three levels (3LSIM) or two levels (2LSIM), and the gradient information of distorted image is segmented by the corresponding regions of reference image, therefore we get multi-level information of these two images. Riesz transform is utilized to get corresponding features of different levels and the corresponding 1st-order and 2nd-order coefficients are combined together by regional mutual information (RMI) and weighted to obtain a single quality score. Experimental results demonstrate that the proposed metric is highly consistent with human subjective evaluations and achieves good performance.     

Image quality assessment based on adaptive multiple Skyline query

Non-reference image quality assessment has attracted great emphasis in recent years. Traditional image quality assessment algorithms based on structural similarity cannot make full use of the image gradient features, and the contrast similarity features often ignore the consistency of continuous color blocks within the image, which leads to large discrepancy between the evaluation result and the subjective judgment of human vision system. In this paper, we propose a deep model for image quality assessment where the spatial and visual features of image are both considered. For better feature fusion, we design an adaptive multiple Skyline query algorithm named MSFF, which takes as input multiple features of images, and learns the feature weights through end-to-end training. Extensive experiments on image quality assessment tasks prove that the proposed model exhibits superior performance compared with existing solutions.     

Visual privacy-preserving level evaluation for multilayer compressed sensing model using contrast and salient structural features

Recognition and classification tasks in images or videos are ubiquitous, but they can lead to privacy issues. People increasingly hope that camera systems can record and recognize important events and objects, such as real-time recording of traffic conditions and accident scenes, elderly fall detection, and in-home monitoring. However, people also want to ensure these activities do not violate the privacy of users or others. The sparse representation classification and recognition algorithms based on compressed sensing (CS) are robust at recognizing human faces from frontal views with varying expressions and illuminations, as well as occlusions and disguises. This is a potential way to perform recognition tasks while preserving visual privacy. In this paper, an improved Gaussian random measurement matrix is adopted in the proposed multilayer CS (MCS) model to realize multiple image CS and achieve a balance between visual privacy-preserving and recognition tasks. The visual privacy-preserving level evaluation for MCS images has important guiding significance for image processing and recognition. Therefore, we propose an image visual privacy-preserving level evaluation method for the MCS model (MCS-VPLE) based on contrast and salient structural features. The basic concept is to use the contrast measurement model based on the statistical mean of the asymmetric alpha-trimmed filter and the salient generalized center-symmetric local binary pattern operator to extract contrast and salient structural features, respectively. The features are fed into a support vector regression to obtain the image quality score, and the fuzzy c-means algorithm is used for clustering to obtain the final evaluated image visual privacy-preserving score. Experiments on three constructed databases show that the proposed method has better prediction effectiveness and performance than conventional methods.     

小波域结构相似度融合图像质量评价方法

针对融合图像质量评价问题,分析了图像质量评价与融合图像质量评价的关系,给出了融合图像质量评价方法的一般表达公式,指出构造实际并不存在的参考图像是解决融合图像质量评价问题的关键。在此基础上,基于空域结构相似度评价方法,对输入源图像和融合图像分别进行小波分解,利用输入源图像小波分解系数构造参考图像小波系数,然后根据人眼视觉敏感度带通特性对参考图像和融合图像的各小波频带进行加权,从而得到整幅图像的小波域结构相似度评价指标,利用目标可探测性、细节可分辨能力和图像整体舒适性构成主观评价指标分别和交互信息量、基于空域的结构相似度比较。实验结果表明,相比于传统的客观评价方法,提出的方法所得结果与主观评价结果的一致性更好。     

Stereo image quality assessment considering the difference of statistical feature in early visual pathway

Human visual theory is closely related to stereo image quality assessment (SIQA), which determines whether the evaluation results of SIQA method can keep good consistency with subjective perception. Many SIQA methods are not fully based on human visual theory, so there is still room for improvement. The research on the visual system tends to the dorsal and ventral pathways, which ignores the information differences in the early visual pathways. It is worth noting that the ON and OFF receptive fields in retinal ganglion cells (RGCs) respond asymmetrically to the statistical features of images. Inspired by this, in this paper, we propose an SIQA method based on monocular and binocular visual features, which takes into account the difference of ON and OFF response features in early visual pathways. Moreover, the different information interaction mechanisms of visual cortex are used to fuse the response maps information of left and right images. Final, monocular and binocular features are extracted and sent to support vector regression (SVR) for quality prediction. Experimental results show that the proposed method is superior to several mainstream SIQA metrics on four publicly available stereo image databases.     

No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks

No-reference image quality assessment is of great importance to numerous image processing applications, and various methods have been widely studied with promising results. These methods exploit handcrafted features in the transformation or space domain that are discriminated for image degradations. However, abundant a priori knowledge is required to extract these handcrafted features. The convolutional neural network (CNN) is recently introduced into the no-reference image quality assessment, which integrates feature learning and regression into one optimization process. Therefore, the network structure generates an effective model for estimating image quality. However, the image quality score obtained by the CNN is based on the mean of all of the image patch scores without considering the human visual system, such as edges and contour of images. In this paper, we combine the CNN and the Prewitt magnitude of segmented images and obtain the image quality score using the mean of all the products of the image patch scores and weights based on the result of segmented images. Experimental results on various image distortion types demonstrate that the proposed algorithm achieves good performance.