Image information and visual quality.

Measurement of visual quality is of fundamental importance to numerous image and video processing applications. The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. Image QA algorithms generally interpret image quality as fidelity or similarity with a "reference" or "perfect" image in some perceptual space. Such "full-reference" QA methods attempt to achieve consistency in quality prediction by modeling salient physiological and psychovisual features of the human visual system (HVS), or by signal fidelity measures. In this paper, we approach the image QA problem as an information fidelity problem. Specifically, we propose to quantify the loss of image information to the distortion process and explore the relationship between image information and visual quality. QA systems are invariably involved with judging the visual quality of "natural" images and videos that are meant for "human consumption." Researchers have developed sophisticated models to capture the statistics of such natural signals. Using these models, we previously presented an information fidelity criterion for image QA that related image quality with the amount of information shared between a reference and a distorted image. In this paper, we propose an image information measure that quantifies the information that is present in the reference image and how much of this reference information can be extracted from the distorted image. Combining these two quantities, we propose a visual information fidelity measure for image QA. We validate the performance of our algorithm with an extensive subjective study involving 779 images and show that our method outperforms recent state-of-the-art image QA algorithms by a sizeable margin in our simulations. The code and the data from the subjective study are available at the LIVE website.     

An information fidelity criterion for image quality assessment using natural scene statistics.

Measurement of visual quality is of fundamental importance to numerous image and video processing applications. The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. Traditionally, image QA algorithms interpret image quality as fidelity or similarity with a "reference" or "perfecft" image in some perceptual space. Such "full-referenc" QA methods attempt to achieve consistency in quality prediction by modeling salient physiological and psychovisual features of the human visual system (HVS), or by arbitrary signal fidelity criteria. In this paper, we approach the problem of image QA by proposing a novel information fidelity criterion that is based on natural scene statistics. QA systems are invariably involved with judging the visual quality of images and videos that are meant for "human consumption." Researchers have developed sophisticated models to capture the statistics of natural signals, that is, pictures and videos of the visual environment. Using these statistical models in an information-theoretic setting, we derive a novel QA algorithm that provides clear advantages over the traditional approaches. In particular, it is parameterless and outperforms current methods in our testing. We validate the performance of our algorithm with an extensive subjective study involving 779 images. We also show that, although our approach distinctly departs from traditional HVS-based methods, it is functionally similar to them under certain conditions, yet it outperforms them due to improved modeling. The code and the data from the subjective study are available at.     

No-reference quality assessment using natural scene statistics: JPEG2000.

Measurement of image or video quality is crucial for many image-processing algorithms, such as acquisition, compression, restoration, enhancement, and reproduction. Traditionally, image quality assessment (QA) algorithms interpret image quality as similarity with a "reference" or "perfect" image. The obvious limitation of this approach is that the reference image or video may not be available to the QA algorithm. The field of blind, or no-reference, QA, in which image quality is predicted without the reference image or video, has been largely unexplored, with algorithms focusing mostly on measuring the blocking artifacts. Emerging image and video compression technologies can avoid the dreaded blocking artifact by using various mechanisms, but they introduce other types of distortions, specifically blurring and ringing. In this paper, we propose to use natural scene statistics (NSS) to blindly measure the quality of images compressed by JPEG2000 (or any other wavelet based) image coder. We claim that natural scenes contain nonlinear dependencies that are disturbed by the compression process, and that this disturbance can be quantified and related to human perceptions of quality. We train and test our algorithm with data from human subjects, and show that reasonably comprehensive NSS models can help us in making blind, but accurate, predictions of quality. Our algorithm performs close to the limit imposed on useful prediction by the variability between human subjects.     

RRED indices: reduced reference entropic differencing for image quality assessment

We study the problem of automatic "reduced-reference" image quality assessment (QA) algorithms from the point of view of image information change. Such changes are measured between the reference- and natural-image approximations of the distorted image. Algorithms that measure differences between the entropies of wavelet coefficients of reference and distorted images, as perceived by humans, are designed. The algorithms differ in the data on which the entropy difference is calculated and on the amount of information from the reference that is required for quality computation, ranging from almost full information to almost no information from the reference. A special case of these is algorithms that require just a single number from the reference for QA. The algorithms are shown to correlate very well with subjective quality scores, as demonstrated on the Laboratory for Image and Video Engineering Image Quality Assessment Database and the Tampere Image Database. Performance degradation, as the amount of information is reduced, is also studied.     

Kurtosis-based no-reference quality assessment of JPEG2000 images

No-reference (NR) image quality assessment (QA) presumes no prior knowledge of reference (distortion-free) images and seeks to quantitatively predict visual quality solely from the distorted images. We develop kurtosis-based NR quality measures for JPEG2000 compressed images in this paper. The proposed measures are based on either 1-D or 2-D kurtosis in the discrete cosine transform (DCT) domain of general image blocks. Comprehensive testing demonstrates their good consistency with subjective quality scores as well as satisfactory performance in comparison with both the representative full-reference (FR) and state-of-the-art NR image quality measures.     

Visual perception of computer-generated stereoscopic pictures: Toward the impact of image resolution

In comparison with the generation of monoscopic images, the time cost of rendering stereoscopic images is doubled. When generating stereoscopic images by computer algorithms, it is desirable to save the computational expense by decreasing the image resolution, without degrading the visual perceptual quality of the images. In this work, to evaluate the perceptual visual quality of computer-generated stereoscopic images (CGSIs), a data set consisting of stereoscopic images created with different horizontal and vertical resolutions was constructed. First, a series of subjective experiments for the analysis of various perceptual situations was conducted. The experimental results show that when the original image resolution was reduced by half, the image difference was not perceptible. In addition, based on full-reference (FR) and no-reference (NR) image quality measurement (IQM), a combined FR-and-NR CGSIQA model was established to predict perceptual quality. We perform weighting calculations for different combinations of FR and NR. The experimental results show that the proposed model significantly outperforms all the classical models.     

Screen content image quality assessment based on convolutional neural networks

Screen content image (SCI) is a composite image including textual and pictorial regions resulting in many difficulties in image quality assessment (IQA). Large SCIs are divided into image patches to increase training samples for CNN training of IQA model, and this brings two problems: (1) local quality of each image patch is not equal to subjective differential mean opinion score (DMOS) of an entire image; (2) importance of different image patches is not same for quality assessment. In this paper, we propose a novel no-reference (NR) IQA model based on the convolutional neural network (CNN) for assessing the perceptual quality of SCIs. Our model conducts two designs solving problems which benefits from two strategies. For the first strategy, to imitate full-reference (FR) CNN-based model behavior, a CNN-based model is designed for both FR and NR IQA, and performance of NR-IQA part improves when the image patch scores predicted by FR-IQA part are adopted as the ground-truth to train NR-IQA part. For the second strategy, image patch qualities of one entire SCI are fused to obtain the SCI quality with an adaptive weighting method taking account the effect of the different image patch contents. Experimental results verify that our model outperforms all test NR IQA methods and most FR IQA methods on the screen content image quality assessment database (SIQAD). On the cross-database evaluation, the proposed method outperforms the existing NR IQA method in terms of at least 2.4 percent in PLCC and 2.8 percent in SRCC, which shows high generalization ability and high effectiveness of our model.     

Quality assessment of multiply and singly distorted stereoscopic images via adaptive construction of cyclopean views

A challenging problem confronted when designing a blind/no-reference (NR) stereoscopic image quality assessment (SIQA) algorithm is to simulate the quality assessment (QA) behavior of the human visual system (HVS) during binocular vision. An effective way to solve this problem is to estimate the quality of the merged single view created in the human brain which is also referred to as the cyclopean image. However, due to the difficulty in modeling the binocular fusion and rivalry properties of the HVS, obtaining effective cyclopean images for QA is non-trivial, and consequently previous NR SIQA algorithms either require the MOS/DMOS values of the distorted 3D images for training or ignore the quality analysis of the merged cyclopean view. In this paper, we focus on (1) constructing accurate and appropriate cyclopean views for QA of stereoscopic images by adaptively analyzing the distortion information of two monocular views, and (2) training NR SIQA models without requiring the assistance of the MOS/DMOS values in existing databases. Accordingly, we present an effective opinion-unaware SIQA algorithm called MUSIQUE-3D, which blindly assesses the quality of multiply and singly distorted stereoscopic images by analyzing quality degradations of both monocular and cyclopean views. The monocular view quality is estimated by an extended version of the MUSIQUE algorithm, and the cyclopean view quality is computed from the distortion parameter values predicted by a two-layer classification-regression model trained on a large 3D image dataset. Tests on various 3D image databases demonstrate the superiority of our method as compared with other state-of-the-art SIQA algorithms.     

Subjective evaluation of stereoscopic image quality

Stereoscopic/3D image and video quality assessment (IQA/VQA) has become increasing relevant in today's world, owing to the amount of attention that has recently been focused on 3D/stereoscopic cinema, television, gaming, and mobile video. Understanding the quality of experience of human viewers as they watch 3D videos is a complex and multi-disciplinary problem. Toward this end we offer a holistic assessment of the issues that are encountered, survey the progress that has been made towards addressing these issues, discuss ongoing efforts to resolve them, and point up the future challenges that need to be focused on. Important tools in the study of the quality of 3D visual signals are databases of 3D image and video sets, distorted versions of these signals and the results of large-scale studies of human opinions of their quality. We explain the construction of one such tool, the LIVE 3D IQA database, which is the first publicly available 3D IQA database that incorporates ‘true’ depth information along with stereoscopic pairs and human opinion scores. We describe the creation of the database and analyze the performance of a variety of 2D and 3D quality models using the new database. The database as well as the algorithms evaluated are available for researchers in the field to use in order to enable objective comparisons of future algorithms. Finally, we broadly summarize the field of 3D QA focusing on key unresolved problems including stereoscopic distortions, 3D masking, and algorithm development.     

No-reference image quality assessment using structural activity

Presuming that human visual perception is highly sensitive to the structural information in a scene, we propose the concept of structural activity (SA) together with a model of SA indicator in a new framework for no-reference (NR) image quality assessment (QA) in this study. The proposed framework estimates image quality based on the quantification of the SA information of different visual significance. We propose some alternative implementations of SA indicator in this paper as examples to demonstrate the effectiveness of the SA-motivated framework. Comprehensive testing demonstrates that the model of SA indicator exhibits satisfactory performance in comparison with subjective quality scores as well as representative full-reference (FR) image quality measures.     

基于互信息量的关键帧全局优化提取方法

关键帧提取技术是视频分析和检索的关键技术之一。提出一种基于互信息量的关键帧提取的新方法,把关键帧提取问题建模为一个多目标规划数学模型,可以同时解决提取关键帧的数量和位置两个主要问题。并且采用了图像信息熵和互信息量作为特征参数参与启发式算法的评价函数,可快速有效地进行子镜头的分割和关键帧的提取。实验证明该方法能较好地提取出视频序列的关键帧。     

Elements of Information Theoretic Methods: Tutorial Introduction

The rudiments of information theoretic methods are introduced and companion papers dealing with solutions of some reliability problems using information theoretic approaches are preluded. Major elements of the communication system are outlined from an information processing point of view. Information is quantified following the work of Shannon. The concepts of uncertainty, self-and mutual information, and entropy are developed as seen at the encoder, channel, and decoder. Channel modeling is demonstrated using a binary symmetric channel as an example. Channel capacity is derived by maximizing transinformation. Elements of coding are described and Shannon's fundamental theorem of discrete noiseless coding is stated. The fundamental relations governing unique decipherability and irreducibility are given and demonstrated by examples. Code efficiency and redundancy are quantified and discussed as system parameters subject to tradeoff. Applications of information theoretic methods in various disciplines are discussed with emphasis on reliability and maintainability. Two unresolved reliability problem areas are identified where, potentially, information theoretic approaches may present a viable solution.     

An optimized CNN-based quality assessment model for screen content image

Most existing convolutional neural network (CNN) based models designed for natural image quality assessment (IQA) employ image patches as training samples for data augmentation, and obtain final quality score by averaging all predicted scores of image patches. This brings two problems when applying these methods for screen content image (SCI) quality assessment. Firstly, SCI contains more complex content compared to natural image. As a result, qualities of SCI patches are different, and the subjective differential mean opinion score (DMOS) is not appropriate as qualities of all image patches. Secondly, the average score of image patches does not represent the quality of entire SCI since the human visual system (HVS) is sensitive to image patches containing texture and edge information. In this paper, we propose a novel quadratic optimized model based on the deep convolutional neural network (QODCNN) for full-reference (FR) and no-reference (NR) SCI quality assessment to overcome these two problems. The contribution of our algorithm can be concluded as follows: 1) Considering the characteristics of SCIs, a valid network architecture is designed for both NR and FR visual quality evaluation of SCIs, which makes the networks learn the feature differences for FR-IQA; 2) with the consideration of correlation between local quality and DMOS, a training data selection method is proposed to fine-tune the pre-trained model with valid SCI patches; 3) an adaptive pooling approach is employed to fuse patch quality to obtain image quality, owns strong noise robust and effects on both FR and NR IQA. Experimental results verify that our model outperforms both current no-reference and full-reference image quality assessment methods on the benchmark screen content image quality assessment database (SIQAD). Cross-database evaluation shows high generalization ability and high effectiveness of our model.     

No-reference image quality assessment based on spatial and spectral entropies

We develop an efficient general-purpose no-reference (NR) image quality assessment (IQA) model that utilizes local spatial and spectral entropy features on distorted images. Using a 2-stage framework of distortion classification followed by quality assessment, we utilize a support vector machine (SVM) to train an image distortion and quality prediction engine. The resulting algorithm, dubbed Spatial–Spectral Entropy-based Quality (SSEQ) index, is capable of assessing the quality of a distorted image across multiple distortion categories. We explain the entropy features used and their relevance to perception and thoroughly evaluate the algorithm on the LIVE IQA database. We find that SSEQ matches well with human subjective opinions of image quality, and is statistically superior to the full-reference (FR) IQA algorithm SSIM and several top-performing NR IQA methods: BIQI, DIIVINE, and BLIINDS-II. SSEQ has a considerably low complexity. We also tested SSEQ on the TID2008 database to ascertain whether it has performance that is database independent.     

Image quality/distortion metric based on α-stable model similarity in wavelet domain

Quality assessment is of central importance in numerous image processing tasks. State-of-the-art objective image quality assessment (IQA) algorithms are generally devised for specific distortion types or based on training procedure of large databases. In this work, we propose a general-purpose full-reference/no-reference (FR/NR) IQA framework for image distortions, nominated by Image Quality/Distortion Metric (IQDM). The leptokurtic and heavy-tailed behaviors of image wavelet coefficients are characterized by symmetric α-stable (SαS) density, and the statistical studies indicate that the model parameters may be altered because of the presence of distortion. This important priori knowledge of original image’s distribution is then used to gauge the distortion between degraded and reference SαS models in multi-scale wavelet sub-bands. We investigate the relationship between original and degraded parameters over scales, accordingly infer the original parameters from the degraded ones. A characteristic probability density function for SαS and its closed-form Kullback–Leibler distance are derived for FR/NR-IQDM using the model parameters. Extensive experiments and comparisons demonstrate that the proposed FR/NR-IQDM scheme is efficacious to most common types of distortion, and leads to a highly comparable performance to the benchmarks and prevalent competitors in consistency with subjective judgements.     

基于信息熵的图像置乱程度评价方法

在基于数字图像的信息隐藏算法中，如何评价图像置乱程度是一个很重要的问题。本文以图像分块信息熵与整体信息熵的比值作为置乱程度的判决标准，并同时利用各分块中互为相邻的像素差值的平均值作为加权系数进行加权平均提出一种新的图像置乱程度的评价方法，实验结果表明该评价方法可以有效地描述图像的置乱程度。     

Memorable and rich video summarization

Video summarization can facilitate rapid browsing and efficient video indexing in many applications. A good summary should maintain the semantic interestingness and diversity of the original video. While many previous methods extracted key frames based on low-level features, this study proposes Memorability-Entropy-based video summarization. The proposed method focuses on creating semantically interesting summaries based on image memorability. Further, image entropy is introduced to maintain the diversity of the summary. In the proposed framework, perceptual hashing-based mutual information (MI) is used for shot segmentation. Then, we use a large annotated image memorability dataset to fine-tune Hybrid-AlexNet. We predict the memorability score by using the fine-tuned deep network and calculate the entropy value of the images. The frame with the maximum memorability score and entropy value in each shot is selected to constitute the video summary. Finally, our method is evaluated on a benchmark dataset, which comes with five human-created summaries. When evaluating our method, we find it generates high-quality results, comparable to human-created summaries and conventional methods.     

No-reference quality assessment for DCT-based compressed image

A blind/no-reference (NR) method is proposed in this paper for image quality assessment (IQA) of the images compressed in discrete cosine transform (DCT) domain. When an image is measured by structural similarity (SSIM), two variances, i.e. mean intensity and variance of the image, are used as features. However, the parameters of original copies are actually unavailable in NR applications; hence SSIM is not widely applicable. To extend SSIM in general cases, we apply Gaussian model to fit quantization noise in spatial domain, and directly estimate noise distribution from the compressed version. Benefit from this rearrangement, the revised SSIM does not require original image as the reference. Heavy compression always results in some zero-value DCT coefficients, which need to be compensated for more accurate parameter estimate. By studying the quantization process, a machine-learning based algorithm is proposed to estimate quantization noise taking image content into consideration. Compared with state-of-the-art algorithms, the proposed IQA is more heuristic and efficient. With some experimental results, we verify that the proposed algorithm (provided no reference image) achieves comparable efficacy to some full reference (FR) methods (provided the reference image), such as SSIM.