Adaptive image coding with perceptual distortion control

This paper presents a discrete cosine transform (DCT)-based locally adaptive perceptual image coder, which discriminates between image components based on their perceptual relevance for achieving increased performance in terms of quality and bit rate. The new coder uses a locally adaptive perceptual quantization scheme based on a tractable perceptual distortion metric. Our strategy is to exploit human visual masking properties by deriving visual masking thresholds in a locally adaptive fashion. The derived masking thresholds are used in controlling the quantization stage by adapting the quantizer reconstruction levels in order to meet the desired target perceptual distortion. The proposed coding scheme is flexible in that it can be easily extended to work with any subband-based decomposition in addition to block-based transform methods. Compared to existing perceptual coding methods, the proposed perceptual coding method exhibits superior performance in terms of bit rate and distortion control. Coding results are presented to illustrate the performance of the presented coding scheme.     

基于纹理分解的变换域JND模型及图像编码方法

为了提高变换域JND模型的精度,在计算对比度掩盖因子时只对纹理分量滤波并判断区域类型的方法避免了JND低估的问题。将改进的JND模型用于图像编码,考虑到辅助信息对编码效率的影响,把经过调整后的JND模型结合到量化过程中,能去除更多的视觉冗余并保持兼容性。仿真结果表明,纹理分解的方法提高了JND阈值,改进的编码方法在相似的视觉质量下能节省更多的码率并且不需要增加额外的比特开销,该编码思路也适用于视频编码。     

Just noticeable distortion model and its applications in video coding

We explore a new perceptually-adaptive video coding (PVC) scheme for hybrid video compression, in order to achieve better perceptual coding quality and operational efficiency. A new just noticeable distortion (JND) estimator for color video is first devised in the image domain. How to efficiently integrate masking effects together is a key issue of JND modelling. We integrate spatial masking factors with the nonlinear additivity model for masking (NAMM). The JND estimator applies to all color components and accounts for the compound impact of luminance masking, texture masking and temporal masking. Extensive subjective viewing confirms that it is capable of determining a more accurate visibility threshold that is close to the actual JND bound in human eyes. Secondly, the image-domain JND profile is incorporated into hybrid video encoding via the JND-adaptive motion estimation and residue filtering process. The scheme works with any prevalent video coding standards and various motion estimation strategies. To demonstrate the effectiveness of the proposed scheme, it has been implemented in the MPEG-2 TM5 coder and demonstrated to achieve average improvement of over 18% in motion estimation efficiency, 0.6 dB in average peak signal-to perceptual-noise ratio (PSPNR) and most remarkably, 0.17 dB in the objective coding quality measure (PSNR) on average. Theoretical explanation is presented for the improvement on the objective coding quality measure. With the JND-based motion estimation and residue filtering process, hybrid video encoding can be more efficient and the use of bits is optimized for visual quality.     

Adaptive Block-size Transform based Just-Noticeable Difference model for images/videos

In this paper, we propose a novel Adaptive Block-size Transform (ABT) based Just-Noticeable Difference (JND) model for images/videos. Extension from 8×8 Discrete Cosine Transform (DCT) based JND model to 16×16 DCT based JND is firstly performed by considering both the spatial and temporal Human Visual System (HVS) properties. For still images or INTRA video frames, a new spatial selection strategy based on the Spatial Content Similarity (SCS) between a macroblock and its sub-blocks is proposed to determine the transform size to be employed to generate the JND map. For the INTER video frames, a temporal selection strategy based on the Motion Characteristic Similarity (MCS) between a macroblock and its sub-blocks is presented to decide the transform size for the JND. Compared with other JND models, our proposed scheme can tolerate more distortions while preserving better perceptual quality. In order to demonstrate the efficiency of the ABT-based JND in modeling the HVS properties, a simple visual quality metric is designed by considering the ABT-based JND masking properties. Evaluating on the image and video subjective databases, the proposed metric delivers a performance comparable to the state-of-the-art metrics. It confirms that the ABT-based JND consists well with the HVS. The proposed quality metric also is applied on ABT-based H.264/Advanced Video Coding (AVC) for the perceptual video coding. The experimental results demonstrate that the proposed method can deliver video sequences with higher visual quality at the same bit-rates.     

基于熵掩蔽的DCT域恰可察觉失真模型

为提高离散余弦变换（discrete cosine transform,DCT）域恰可察觉失真（just noticeable distortion,JND）模型阈值精度并避免跨域操作，将熵掩蔽效应引入DCT域JND模型。首先，从自由能理论和贝叶斯推理出发，设计基于DCT域纹理能量相似性的自回归模型模拟视觉感知过程中的自发预测行为；其次，探索视觉感知与预测残差的映射关系得到块级无序度，并将熵掩蔽效应建模为关于无序度的JND阈值调节因子；最后，结合空间对比敏感度函数、亮度自适应掩蔽以及对比度掩蔽，提出基于熵掩蔽的DCT域JND模型。与现有DCT域JND模型相比，所提模型所有运算均在DCT域执行，更高效简洁。主观、客观实验结果表明，所提模型在感知质量相同或更好的情况下，噪声污染图的平均峰值信噪比（peak signal-to-noise ratio,PSNR）值比其他4个JND对比模型低2.04 dB，更符合人眼视觉系统的感知特性。     

Perceptually adaptive joint deringing–deblocking filtering for scalable video transmission over wireless networks

Video transmission over low bit-rate channels, such as wireless networks, requires dedicated filtering during decoding for crucial enhancement of the perceptual video quality. For that reason, deringing and deblocking are inevitable components of decoders in wireless video transmission systems. Aimed at improving the visual quality of decoded video, in this paper a new perceptually adaptive joint deringing–deblocking filtering technique for scalable video streams is introduced. The proposed approach is designed to deal with artefacts inherent to transmissions over very low bit-rate channels, specifically wireless networks. It considers both prediction and update steps in motion compensated temporal filtering in an in-loop filtering architecture. The proposed approach integrates three different filtering modules to deal with low-pass, high-pass and after-update frames, respectively. The filter strength is adaptively tuned according to the number of discarded bit-planes, which in turn depends on the channel bit-rate and the channel error conditions. Furthermore, since ringing and blocking artefacts are visually annoying, relevant characteristics of the human visual system are considered in the used bilateral filtering model. That is, the amount of filtering is adjusted to the perceptual distortion by integrating a human visual system model into filtering based on luminance, activity and temporal masking. As a consequence, the resulting filter strength is automatically adapted to both perceptual sensitivity and channel variation. To assess the performance of the proposed approach, a comprehensive comparative evaluation against the conventional loop architecture and bilateral filter was conducted. The results of the experimental evaluation show a superior performance of the proposed adaptive filtering approach, providing better objective and subjective quality.     

Adaptive progressive filter to remove impulse noise in highly corrupted color images

In this paper, an adaptive progressive filtering (APF) technique with low computational complexity is proposed for removing impulse noise in highly corrupted color images. Color images that are corrupted with impulse noise are generally filtered by applying a vector-based approach. Vector-based methods tend to cluster the noise and receive a lower noise reduction performance when the noise ratio is high. To improve the performance, in the proposed technique, a new reliable estimation of impulse noise intensity and noise type is made initially, and then a progressive restoration mechanism is devised, using multi-pass non-linear operations with selected processing windows adapted to the estimation. The effect of impulse detection based on geometric characteristics and features of the corrupt pixel/pixel regions and the exact estimation of impulse noise intensity and type are used in the APF to efficiently support the progressive filtering mechanism. Through experiments conducted using a range of color images, the proposed filtering technique has demonstrated superior performance to that of well-known benchmark techniques, in terms of standard objective measurements, visual image quality, and the computational complexity.     

Geometric Features-Based Filtering for Suppression of Impulse Noise in Color Images

A geometric features-based filtering technique, named as the adaptive geometric features based filtering technique (AGFF), is presented for removal of impulse noise in corrupted color images. In contrast with the traditional noise detection techniques where only 1-D statistical information is used for noise detection and estimation, a novel noise detection method is proposed based on geometric characteristics and features (i.e., the 2-D information) of the corrupted pixel or the pixel region, leading to effective and efficient noise detection and estimation outcomes. A progressive restoration mechanism is devised using multipass nonlinear operations which adapt to the intensity and the types of the noise. Extensive experiments conducted using a wide range of test color images have shown that the AGFF is superior to a number of existing well-known benchmark techniques, in terms of standard image restoration performance criteria, including objective measurements, the visual image quality, and the computational complexity.      

Partition-based vector filtering technique for suppression of noise in digital color images.

A partition-based adaptive vector filter is proposed for the restoration of corrupted digital color images. The novelty of the filter lies in its unique three-stage adaptive estimation. The local image structure is first estimated by a series of center-weighted reference filters. Then the distances between the observed central pixel and estimated references are utilized to classify the local inputs into one of preset structure partition cells. Finally, a weighted filtering operation, indexed by the partition cell, is applied to the estimated references in order to restore the central pixel value. The weighted filtering operation is optimized off-line for each partition cell to achieve the best tradeoff between noise suppression and structure preservation. Recursive filtering operation and recursive weight training are also investigated to further boost the restoration performance. The proposed filter has demonstrated satisfactory results in suppressing many distinct types of noise in natural color images. Noticeable performance gains are demonstrated over other prior-art methods in terms of standard objective measurements, the visual image quality and the computational complexity.     

Locally adaptive perceptual image coding

Most existing efforts in image and video compression have focused on developing methods to minimize not perceptual but rather mathematically tractable, easy to measure, distortion metrics. While nonperceptual distortion measures were found to be reasonably reliable for higher bit rates (high-quality applications), they do not correlate well with the perceived quality at lower bit rates and they fail to guarantee preservation of important perceptual qualities in the reconstructed images despite the potential for a good signal-to-noise ratio (SNR). This paper presents a perceptual-based image coder, which discriminates between image components based on their perceptual relevance for achieving increased performance in terms of quality and bit rate. The new coder is based on a locally adaptive perceptual quantization scheme for compressing the visual data. Our strategy is to exploit human visual masking properties by deriving visual masking thresholds in a locally adaptive fashion based on a subband decomposition. The derived masking thresholds are used in controlling the quantization stage by adapting the quantizer reconstruction levels to the local amount of masking present at the level of each subband transform coefficient. Compared to the existing non-locally adaptive perceptual quantization methods, the new locally adaptive algorithm exhibits superior performance and does not require additional side information. This is accomplished by estimating the amount of available masking from the already quantized data and linear prediction of the coefficient under consideration. By virtue of the local adaptation, the proposed quantization scheme is able to remove a large amount of perceptually redundant information. Since the algorithm does not require additional side information, it yields a low entropy representation of the image and is well suited for perceptually lossless image compression.     

Postprocessing of low bit-rate block DCT coded images based on a fields of experts prior.

Transform coding using the discrete cosine transform (DCT) has been widely used in image and video coding standards, but at low bit rates, the coded images suffer from severe visual distortions which prevent further bit reduction. Postprocessing can reduce these distortions and alleviate the conflict between bit rate reduction and quality preservation. Viewing postprocessing as an inverse problem, we propose to solve it by the maximum a posteriori criterion. The distortion caused by coding is modeled as additive, spatially correlated Gaussian noise, while the original image is modeled as a high order Markov random field based on the fields of experts framework. Experimental results show that the proposed method, in most cases, achieves higher PSNR gain than other methods and the processed images possess good visual quality. In addition, we examine the noise model used and its parameter setting. The noise model assumes that the DCT coefficients and their quantization errors are independent. This assumption is no longer valid when the coefficients are truncated. We explain how this problem can be rectified using the current parameter setting.     

基于感知加权滤波的卡尔曼滤波语音增强

结合人耳听觉掩蔽效应,提出一种基于听觉感知加权的卡尔曼滤波语音增强方法。由于人耳对语音的感知主要是通过语音信号频谱分量幅度获得的,引入听觉感知加权滤波器在频域上使共振峰区域残留噪声更多,而共振峰之间及语音幅度谱较低的区域残留噪声减少,这样符合人耳的听觉特性,从而使得主观感觉到的噪声最小。采用语音质量感知评估对语音增强的效果进行评测,与传统的卡尔曼滤波语音增强算法相比,实验结果显示该算法提高了增强语音的质量。     

Content adaptive video denoising based on human visual perception

In this paper, we propose content adaptive denoising in highly corrupted videos based on human visual perception. We introduce the human visual perception in video denoising to achieve good performance. In general, smooth regions corrupted by noise are much more annoying to human observers than complex regions. Moreover, human eyes are more interested in complex regions with image details and more sensitive to luminance than chrominance. Based on the human visual perception, we perform perceptual video denoising to effectively preserve image details and remove annoying noise. To successfully remove noise and recover the image details, we extend nonlocal mean filtering to the spatiotemporal domain. With the guidance of content adaptive segmentation and motion detection, we conduct content adaptive filtering in the YUV color space to consider context in images and obtain perceptually pleasant results. Extensive experiments on various video sequences demonstrate that the proposed method reconstructs natural-looking results even in highly corrupted images and achieves good performance in terms of both visual quality and quantitative measures.     

Adaptive cosine transform image coding with constant blockdistortion

An adaptive block discrete-cosine transform (DCT) coding scheme is implemented with the same average distortion designated for each block. This constant distortion designation not only has perceptual advantages, but also allows the rate to vary, adjusting to the changing spectral characteristics among the blocks. The successful execution of this scheme requires a different spectral estimate for each block. To keep overhead and computation within limits, a novel technique is introduced by which a two-dimensional block spectrum is characterized by a one-dimensional autoregressive model. Simulations with images of natural scenes and medical radiology provide reconstructions with nearly uniform block distortion and very high visual and measurable quality at low rates     

基于局部自适应色差阈值的彩色图像边缘检测

为了使彩色图像的边缘检测器更符合人眼对图像信息的分辨情况,防止视觉不敏感区域的边缘的过检测问题,该文提出一种自适应色差阈值的估计方法并与不同的色彩梯度算子结合应用于彩色图像的边缘检测中。构建包括亮度掩模与对比灵敏度的局部色差可视阈值的权重因子,结合局部背景亮度以及亮度与色彩的空间频率对人眼视觉的影响。利用信噪比(SNR), Pratt因子与时间复杂度对提出的算法的抗噪性与边缘定位的准确性以及时间代价进行定量评价,表明该算法能准确检测出图像边缘且有效地抵抗噪声对图像的干扰。     

自适应的混合域音频水印新算法

本文结合小波包变换和离散余弦变换，提出了一种基于听觉模型的混合域自适应音频盲水印算法，在不引入听觉失真的前提下，实现了自适应的水印嵌入。算法首先对音频信号进行小波包分解，使得分解后的子带更接近人耳临界频带。其次对每个子带的小波包系数进行离散余弦变换，计算出子带掩蔽阈值。根据子带掩蔽阈值自适应的选取噪声敏感度小的音频段作为水印嵌入段，选取功率值低于掩蔽阈值的频域系数作为水印嵌入位置，同时采用噪声掩蔽比调整水印嵌入强度。二值水印图像通过量化索引调制的方法嵌入到音频信号的中低频系数中，提取水印时不需要原始音频载体。本算法在水印容量、不可感知性和鲁棒性之间达到了很好的平衡，水印容量在576.7bps到689.5bps之间，算法对添加噪声、重新量化、重新采样、低通滤波和MP3压缩均具有很好的鲁棒性。      

Just-noticeable difference estimation with pixels in images

Perceptual visibility threshold estimation, based upon characteristics of the human visual system (HVS), is widely used in digital image and video processing. We propose in this paper a scheme for estimating JND (just-noticeable difference) with explicit formulation for image pixels, by summing the effects of the visual thresholds in sub-bands. The factors being considered include spatial contrast sensitivity function (CSF), luminance adaptation, and adaptive inter- and intra-band contrast masking. The proposed scheme demonstrates favorable results in noise shaping and perceptual visual distortion gauge for different images, in comparison with the relevant existing JND estimators.     

Learning-based JNCD prediction for quality-wise perceptual quantization in HEVC

In visual perception, human only perceive discrete-scale quality levels over a wide range of coding bitrate. More clearly, the videos compressed with a series of quantization parameters (QPs) only have limited perceived quality levels. In this paper, perceptual quantization is transformed into the problem of how to determine the just perceived QP for each quality level, and a just noticeable coding distortion (JNCD) based perceptual quantization scheme is proposed. Specifically, multiple visual masking effects are analyzed and a linear regression (LR) based JNCD model is proposed to predict JNCD thresholds for all quality levels at first. According to the JNCD prediction model, the frame-level perceptual QPs for all quality levels are then derived on the premise of that coding distortions are infinitely close to the predicted JNCD thresholds. Based on the predicted frame-level perceptual QPs, the perceived QPs of all quality levels for each coding unit (CU) are finally determined according to a perceptual modulation function. Experimental results show that the proposed quality-wise perceptual quantization scheme is superior to the existing perceptual video coding algorithms significantly, i.e., the proposed perceptual quantization could save more bitrate with better quality.     

Design a deblocking filter with three separate modes in DCT-based coding

The reconstructed images from highly compressed data have noticeable image degradations, such as blocking artifacts near the block boundaries. Post-processing appears to be the most feasible solution because it does not require any existing standards to be changed. Markedly reducing blocking effects can increase compression ratios for a particular image quality or improve the quality of equally compressed images. In this work, a novel deblocking algorithm is proposed based on three filtering modes in terms of the activity across block boundaries. By properly considering the masking effect of the HVS (Human Visual System), an adaptive filtering decision is integrated into the deblocking process. According to three different deblocking modes appropriate for local regions with different characteristics, the perceptual and objective quality are improved without excessive smoothing the image details or insufficiently reducing the strong blocking effect on a flat region. According to the simulation results, the proposed method outperforms other deblocking algorithms in respect to PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural SIMilarity).