Removing label ambiguity in learning-based visual saliency estimation

Visual saliency is a useful clue to depict visually important image/video contents in many multimedia applications. In visual saliency estimation, a feasible solution is to learn a "feature-saliency" mapping model from the user data obtained by manually labeling activities or eye-tracking devices. However, label ambiguities may also arise due to the inaccurate and inadequate user data. To process the noisy training data, we propose a multi-instance learning to rank approach for visual saliency estimation. In our approach, the correlations between various image patches are incorporated into an ordinal regression framework. By iteratively refining a ranking model and relabeling the image patches with respect to their mutual correlations, the label ambiguities can be effectively removed from the training data. Consequently, visual saliency can be effectively estimated by the ranking model, which can pop out real targets and suppress real distractors. Extensive experiments on two public image data sets show that our approach outperforms 11 state-of-the-art methods remarkably in visual saliency estimation.     

An integration of bottom-up and top-down salient cues on RGB-D data: saliency from objectness versus non-objectness

Bottom-up and top-down visual cues are two types of information that helps the visual saliency models. These salient cues can be from spatial distributions of the features (space-based saliency) or contextual/task-dependent features (object-based saliency). Saliency models generally incorporate salient cues either in bottom-up or top-down norm separately. In this work, we combine bottom-up and top-down cues from both space- and object-based salient features on RGB-D data. In addition, we also investigated the ability of various pre-trained convolutional neural networks for extracting top-down saliency on color images based on the object dependent feature activation. We demonstrate that combining salient features from color and dept through bottom-up and top-down methods gives significant improvement on the salient object detection with space-based and object-based salient cues. RGB-D saliency integration framework yields promising results compared with the several state-of-the-art-models.     

High-level background prior based salient object detection

Salient object detection is a fundamental problem in computer vision. Existing methods using only low-level features failed to uniformly highlight the salient object regions. In order to combine high-level saliency priors and low-level appearance cues, we propose a novel Background Prior based Salient detection method (BPS) for high-quality salient object detection.Different from other background prior based methods, a background estimation is added before performing saliency detection. We utilize the distribution of bounding boxes generated by a generic object proposal method to obtain background information. Three background priors are mainly considered to model the saliency, namely background connectivity prior, background contrast prior and spatial distribution prior, allowing the proposed method to highlight the salient object as a whole and suppress background clutters.Experiments conducted on two benchmark datasets validate that our method outperforms 11 state-of-the-art methods, while being more efficient than most leading methods.     

基于深度学习的图像显著对象检测

显著区域检测可应用在对象识别、图像分割、视 频/图像压缩中,是计算机视觉领域的重要研究主题。然而,基于不 同视觉显著特征的显著区域检测法常常不能准确地探测出显著对象且计算费时。近来,卷积 神经网络模型在图像分析和处理 领域取得了极大成功。为提高图像显著区域检测性能,本文提出了一种基于监督式生成对抗 网络的图像显著性检测方法。它 利用深度卷积神经网络构建监督式生成对抗网络,经生成器网络与鉴别器网络的不断相互对 抗训练,使卷积网络准确学习到 图像显著区域的特征,进而使生成器输出精确的显著对象分布图。同时,本文将网络自身误 差和生成器输出与真值图间的 L1距离相结合,来定义监督式生成对抗网络的损失函数,提升了显著区域检测精度。在MSRA 10K与ECSSD数据库上的实 验结果表明,本文方法 分别获得了94.19%与96.24%的准确率和93.99%与90.13%的召回率,F -Measure值也高达94.15%与94.76%,优于先 前常用的显著性检测模型。     

Superpixel based color contrast and color distribution driven salient object detection

Color is the most informative low-level feature and might convey tremendous saliency information of a given image. Unfortunately, color feature is seldom fully exploited in the previous saliency models. Motivated by the three basic disciplines of a salient object which are respectively center distribution prior, high color contrast to surroundings and compact color distribution, in this paper, we design a comprehensive salient object detection system which takes the advantages of color contrast together with color distribution and outputs high quality saliency maps. The overall procedure flow of our unified framework contains superpixel pre-segmentation, color contrast and color distribution computation, combination, and final refinement.In color contrast saliency computation, we calculate center-surrounded color contrast and then employ the distribution prior in order to select correct color components. A global saliency smoothing procedure that is based on superpixel regions is introduced as well. This processing step preferably alleviates the saliency distortion problem, leading to the entire object being highlighted uniformly. Finally, a saliency refinement approach is adopted to eliminate artifacts and recover unconnected parts within the combined saliency maps.In visual comparison, our method produces higher quality saliency maps which stress out the total object meanwhile suppress background clutter. Both qualitative and quantitative experiments show our approach outperforms 8 state-of-the-art methods, achieving the highest precision rate 96% (3% improvement from the current highest), when evaluated via one of the most popular data sets. Excellent content-aware image resizing also could be achieved using our saliency maps.     

Bayesian salient object detection based on saliency driven clustering

Salient object detection is essential for applications, such as image classification, object recognition and image retrieval. In this paper, we design a new approach to detect salient objects from an image by describing what does salient objects and backgrounds look like using statistic of the image. First, we introduce a saliency driven clustering method to reveal distinct visual patterns of images by generating image clusters. The Gaussian Mixture Model (GMM) is applied to represent the statistic of each cluster, which is used to compute the color spatial distribution. Second, three kinds of regional saliency measures, i.e, regional color contrast saliency, regional boundary prior saliency and regional color spatial distribution, are computed and combined. Then, a region selection strategy integrating color contrast prior, boundary prior and visual patterns information of images is presented. The pixels of an image are divided into either potential salient region or background region adaptively based on the combined regional saliency measures. Finally, a Bayesian framework is employed to compute the saliency value for each pixel taking the regional saliency values as priority. Our approach has been extensively evaluated on two popular image databases. Experimental results show that our approach can achieve considerable performance improvement in terms of commonly adopted performance measures in salient object detection.     

A heuristic framework for perceptual saliency prediction

Saliency prediction can be regarded as the human spontaneous activity. The most effective saliency model should highly approximate the response of viewers to the perceived information. In the paper, we exploit the perception response for saliency detection and propose a heuristic framework to predict salient region. First, to find the perceptually meaningful salient regions, an orientation selectivity based local feature and a visual Acuity based global feature are proposed to jointly predict candidate salient regions. Subsequently, to further boost the accuracy of saliency map, we introduce a visual error sensitivity based operator to activate the meaningful salient regions from a local and global perspective. In addition, an adaptive fusion method based on free energy principle is designed to combine the sub-saliency maps from each image channel to obtain the final saliency map. Experimental results on five natural and emotional datasets demonstrate the superiority of the proposed method compared to twelve state-of-the-art algorithms.     

Global salient information maximization for saliency detection

In this paper, a new method for saliency detection is proposed. Based on the defined features of the salient object, we solve the problem of saliency detection from three aspects. Firstly, from the view of global information, we partition the image into two clusters, namely, salient component and background component by employing Principal Component Analysis (PCA) and k-means clustering. Secondly, the maximal salient information is applied to find the position of saliency and eliminate the noise. Thirdly, we enhance the saliency for the salient regions while weaken the background regions. Finally, the saliency map is obtained based on these aspects. Experimental results show that the proposed method achieves better results than the state of the art methods. And this method can be applied for graph based salient object segmentation.     

视觉注意驱动的基于混沌分析的运动检测方法

提出了视觉注意驱动的基于混沌分析的运动检测方法(MDSA)。MDSA首先基于视觉注意机制提取图像的显著区域,而后对显著区域进行混沌分析以检测运动目标。算法技术路线为:首先根据场景图像提取多种视觉敏感的底层图像特征;然后根据特征综合理论将这些特征融合起来得到一幅反映场景图像中各个位置视觉显著性的显著图;而后对显著性水平最高的图像位置所在的显著区域运用混沌分析的方法进行运动检测;根据邻近优先和返回抑制原则提取下一最显著区域并进行运动检测,直至遍历所有的显著区域。本文对传统的显著区域提取方法进行了改进以减少计算量:以邻域标准差代替center-surround算子评估图像各位置的局部显著度,采用显著点聚类的方法代替尺度显著性准则提取显著区域;混沌分析首先判断各显著区域的联合直方图（JH）是否呈现混沌特征,而后依据分维数以一固定阈值对存在混沌的JH中各散点进行分类,最后将分类结果对应到显著区域从而实现运动分割。MDSA具有较好的运动分割效果和抗噪性能,对比实验和算法开销分析证明MDSA优于基于马塞克的运动检测方法（MDM）。      

超像素内容感知先验的多尺度贝叶斯显著性检测方法

针对复杂背景下显著性检测方法不能够有效地抑制背景,进而准确地检测目标这一问题,提出了超像素内容感知先验的多尺度贝叶斯显著性检测方法.首先,将目标图像分割为多尺度的超像素图,在每个尺度上引入内容感知的对比度先验、中心位置先验、边界连通背景先验来计算单一尺度上的目标显著值;其次,融合多个尺度的内容感知先验显著值生成一个粗略的显著图;然后,将粗略显著图值作为先验概率,根据颜色直方图和凸包中心先验计算观测似然概率,再使用多尺度贝叶斯模型来获取最终显著目标;最后,使用了3个公开的数据集、5种评估指标、7种现有的方法进行对比实验,结果表明本文方法在显著性目标检测方面具有更好的表现.     

A novel video salient object extraction method based on visual attention

This paper addresses a novel approach to automatically extract video salient objects based on visual attention mechanism and seeded object growing technique. First, a dynamic visual attention model to capture the object motions by global motion estimation and compensation is constructed. Through combining it with a static attention model, a saliency map is formed. Then, with a modified inhibition of return (MIOR) strategy, the winner-take-all (WTA) neural network is used to scan the saliency map for the most salient locations selected as attention seeds. Lastly, the particle swarm optimization (PSO) algorithm is employed to grow the attention objects modeled by Markov random field (MRF) from the seeds. Experiments verify that our presented approach could extract both of stationary and moving salient objects efficiently.     

RLP-AGMC: Robust label propagation for saliency detection based on an adaptive graph with multiview connections

There have been remarkable improvements in the salient object detection in the recent years. During the past few years, graph-based saliency detection algorithms have been proposed and made advances. Nevertheless, most of the state-of-the-art graph-based approaches are usually designed with low-level features, misleading assumption, fixed predefined graph structure and weak affinity matrix, which determine that they are not robust enough to handle images with complex or cluttered background.In this paper, we propose a robust label propagation-based mechanism for salient object detection throughout an adaptive graph to tackle above issues. Low-level features as well as deep features are integrated into the proposed framework to measure the similarity between different nodes. In addition, a robust mechanism is presented to calculate seeds based on the distribution of salient regions, which can achieve desirable results even if the object is in contact with the image boundary and the image scene is complex. Then, an adaptive graph with multiview connections is constructed based on different cues to learn the graph affinity matrix, which can better capture the characteristics between spatially adjacent and distant regions. Finally, a novel RLP-AGMC model, i.e. robust label propagation throughout an adaptive graph with multiview connections, is put forward to calculate saliency maps in combination with the obtained seed vectors. Comprehensive experiments on six public datasets demonstrate the proposed method outperforms fourteen existing state-of-the-art methods in terms of various evaluation metrics.     

Saliency detection based on singular value decomposition

Saliency detection has gained popularity in many applications, and many different approaches have been proposed. In this paper, we propose a new approach based on singular value decomposition (SVD) for saliency detection. Our algorithm considers both the human-perception mechanism and the relationship between the singular values of an image decomposed by SVD and its salient regions. The key concept of our proposed algorithms is based on the fact that salient regions are the important parts of an image. The singular values of an image are divided into three groups: large, intermediate, and small singular values. We propose the hypotheses that the large singular values mainly contain information about the non-salient background and slight information about the salient regions, while the intermediate singular values contain most or even all of the saliency information. The small singular values contain little or even none of the saliency information. These hypotheses are validated by experiments. By regularization based on the average information, regularization using the leading largest singular values or regularization based on machine learning, the salient regions will become more conspicuous. In our proposed approach, learning-based methods are proposed to improve the accuracy of detecting salient regions in images. Gaussian filters are also employed to enhance the saliency information. Experimental results prove that our methods based on SVD achieve superior performance compared to other state-of-the-art methods for human-eye fixations, as well as salient-object detection, in terms of the area under the receiver operating characteristic (ROC) curve (AUC) score, the linear correlation coefficient (CC) score, the normalized scan-path saliency (NSS) score, the F-measure score, and visual quality.     

基于对比敏感度的小波域图像显著性检测

A method of high definition saliency detection based on contrast sensitive function and wavelet analysis was proposed in order to improve the resolution of saliency maps.Original image was filtered by contrast sensitive function in YCbCr space,which could simulate the contrast of human eyes; then wavelet decomposition was carried out in Y,Cb,and Cr three channels individually,low frequency and high frequency feature saliency maps were extracted and further combined to obtain saliency map in single channel; finally saliency maps in three channels were fused to the high resolution saliency map.Experiments result show that the saliency images have high resolution,well-defined boundaries,and whole highlight salient objects.     

Are emotional objects visually salient? The Emotional Maps Database

The visual system prioritizes emotional content in natural scenes, but it is unclear whether emotional objects are systematically more salient. We compare emotional maps - created by averaging multiple manual selections of the most meaningful regions in images of negative, positive, and neutral affective valence - with saliency maps generated by Graph-Based Visual Saliency, Proto-object, and SalGAN models. We found that similarity between emotional and saliency maps is modulated by the scenes’ arousal and valence ratings: the more negative and high-arousing content, the less it was salient. Simultaneously, the negative and high-arousing content was the easiest to identify by the participants, as shown by the highest inter-individual agreement in the selections. Our results support the “affective gap” hypothesis, i.e., decoupling of emotional meaning from image’s formal features. The Emotional Maps Database created for this study, proven useful in gaze fixation prediction, is available online for scientific use.     

Video viewing: do auditory salient events capture visual attention?

We assess whether salient auditory events contained in soundtracks modify eye movements when exploring videos. In a previous study, we found that, on average, nonspatial sound contained in video soundtracks impacts on eye movements. This result indicates that sound could play a leading part in visual attention models to predict eye movements. In this research, we go further and test whether the effect of sound on eye movements is stronger just after salient auditory events. To automatically spot salient auditory events, we used two auditory saliency models: the discrete energy separation algorithm and the energy model. Both models provide a saliency time curve, based on the fusion of several elementary audio features. The most salient auditory events were extracted by thresholding these curves. We examined some eye movement parameters just after these events rather than on all the video frames. We showed that the effect of sound on eye movements (variability between eye positions, saccade amplitude, and fixation duration) was not stronger after salient auditory events than on average over entire videos. Thus, we suggest that sound could impact on visual exploration not only after salient events but in a more global way.     

HTRI: High time range imaging

HDRI (High Dynamic Range Imaging) techniques are used to produce dynamic and well-contrasted images of real-world luminance, by capturing several images of the same scene through exposure bracketing. In the same vein, we have developed a new approach to image fusion from a series of photographs of the same scene taken at different timestamps. When compared with HDRI, exposure bracketing at a single timestamp is replaced by timestamp variation disregarding exposure times. Because of the parallel between these two approaches, this technique is called HTRI (High Time Range Imaging), it aims at capturing ephemeral events occurring over a long time period during which a sequence of images is shot.For each pixel location, the most salient colors are privileged in the series of photographs. The choice of the saliency criterion is based on an analysis of the existing admitted definitions of visual attention. In a second stage, a higher priority is assigned to the pixels with high temporal saliency, i.e., which appear very briefly in the sequence, jointly producing spatial and temporal changes of contrast between two successive frames. The proposed algorithm captures all these salient objects in the final image, without introducing a significant amount of noise, and despite the large illumination changes that may occur in the acquisition conditions from one frame to the next. Experiments evaluate the impact of the method parameters, and confirm benefits of HTRI compared to other fusion techniques.     

Video saliency detection incorporating temporal information in compressed domain

Saliency detection is widely used to pick out relevant parts of a scene as visual attention regions for various image/video applications. Since video is increasingly being captured, moved and stored in compressed form, there is a need for detecting video saliency directly in compressed domain. In this study, a compressed video saliency detection algorithm is proposed based on discrete cosine transformation (DCT) coefficients and motion information within a visual window. Firstly, DCT coefficients and motion information are extracted from H.264 video bitstream without full decoding. Due to a high quantization parameter setting in encoder, skip/intra is easily chosen as the best prediction mode, resulting in a large number of blocks with zero motion vector and no residual existing in video bitstream. To address these problems, the motion vectors of skip/intra coded blocks are calculated by interpolating its surroundings. In addition, a visual window is constructed to enhance the contrast of features and to avoid being affected by encoder. Secondly, after spatial and temporal saliency maps being generated by the normalized entropy, a motion importance factor is imposed to refine the temporal saliency map. Finally, a variance-like fusion method is proposed to dynamically combine these maps to yield the final video saliency map. Experimental results show that the proposed approach significantly outperforms other state-of-the-art video saliency detection models.     

基于虚拟光学的视觉显著目标可控放大重建

该文提出一种基于虚拟光学的视觉显著目标高分辨率可控放大重建方法。原始图像放置于虚拟光路物平面,首先通过衍射逆计算获得原始图像在虚拟衍射面的光波信号,再对虚拟衍射面光波用球面波照射后作正向衍射计算,通过改变观测平面位置可重建出不同放大率的原始图像。仿真测试结果表明,与一般的插值放大方法相比,所获得的放大后的图像特别是在显著性区域表示出良好的视觉感知效果。将包含人脸的低分辨率降质图像作为待重建信号,所重建人脸的显著性区域如眼睛、鼻子等比一般重建方法更清晰。用水平集方法结合显著图分割出原始图像中的局部显著区域并作放大重建和轮廓提取,轮廓表现出良好的光滑性。