Bayesian salient object detection based on saliency driven clustering

Salient object detection is essential for applications, such as image classification, object recognition and image retrieval. In this paper, we design a new approach to detect salient objects from an image by describing what does salient objects and backgrounds look like using statistic of the image. First, we introduce a saliency driven clustering method to reveal distinct visual patterns of images by generating image clusters. The Gaussian Mixture Model (GMM) is applied to represent the statistic of each cluster, which is used to compute the color spatial distribution. Second, three kinds of regional saliency measures, i.e, regional color contrast saliency, regional boundary prior saliency and regional color spatial distribution, are computed and combined. Then, a region selection strategy integrating color contrast prior, boundary prior and visual patterns information of images is presented. The pixels of an image are divided into either potential salient region or background region adaptively based on the combined regional saliency measures. Finally, a Bayesian framework is employed to compute the saliency value for each pixel taking the regional saliency values as priority. Our approach has been extensively evaluated on two popular image databases. Experimental results show that our approach can achieve considerable performance improvement in terms of commonly adopted performance measures in salient object detection.     

An integrated region-, boundary-, shape-based active contour for multiple object overlap resolution in histological imagery

Active contours and active shape models (ASM) have been widely employed in image segmentation. A major limitation of active contours, however, is in their 1) inability to resolve boundaries of intersecting objects and to 2) handle occlusion. Multiple overlapping objects are typically segmented out as a single object. On the other hand, ASMs are limited by point correspondence issues since object landmarks need to be identified across multiple objects for initial object alignment. ASMs are also are constrained in that they can usually only segment a single object in an image. In this paper, we present a novel synergistic boundary and region-based active contour model that incorporates shape priors in a level set formulation with automated initialization based on watershed. We demonstrate an application of these synergistic active contour models using multiple level sets to segment nuclear and glandular structures on digitized histopathology images of breast and prostate biopsy specimens. Unlike previous related approaches, our model is able to resolve object overlap and separate occluded boundaries of multiple objects simultaneously. The energy functional of the active contour is comprised of three terms. The first term is the prior shape term, modeled on the object of interest, thereby constraining the deformation achievable by the active contour. The second term, a boundary-based term detects object boundaries from image gradients. The third term drives the shape prior and the contour towards the object boundary based on region statistics. The results of qualitative and quantitative evaluation on 100 prostate and 14 breast cancer histology images for the task of detecting and segmenting nuclei and lymphocytes reveals that the model easily outperforms two state of the art segmentation schemes (geodesic active contour and Rousson shape-based model) and on average is able to resolve up to 91% of overlapping/occluded structures in the images.     

Hand pose estimation in object-interaction based on deep learning for virtual reality applications

Hand Pose Estimation aims to predict the position of joints on a hand from an image, and it has become popular because of the emergence of VR/AR/MR technology. Nevertheless, an issue surfaces when trying to achieve this goal, since a hand tends to cause self-occlusion or external occlusion easily as it interacts with external objects. As a result, there have been many projects dedicated to this field for a better solution of this problem. This paper develops a system that accurately estimates a hand pose in 3D space using depth images for VR applications. We propose a data-driven approach of training a deep learning model for hand pose estimation with object interaction. In the convolutional neural network (CNN) training procedure, we design a skeleton-difference loss function, which effectively can learn the physical constraints of a hand. Also, we propose an object-manipulating loss function, which considers knowledge of the hand-object interaction, to enhance performance.In the experiments we have conducted for hand pose estimation under different conditions, the results validate the robustness and the performance of our system and show that our method is able to predict the joints more accurately in challenging environmental settings. Such appealing results may be attributed to the consideration of the physical joint relationship as well as object information, which in turn can be applied to future VR/AR/MR systems for more natural experience.     

Microwave image reconstruction utilizing log-magnitude and unwrapped phase to improve high-contrast object recovery

Reconstructing images of large high-contrast objects with microwave methods has proved difficult. Successful images have generally been obtained by using a priori information to constrain the image reconstruction to recover the correct electromagnetic property distribution. In these situations, the measured electric field phases as a function of receiver position around the periphery of the imaging field-of-view vary rapidly often undergoing changes of greater than pi radians especially when the object contrast and illumination frequency increase. In this paper, we introduce a modified form of a Maxwell equation model-based image reconstruction algorithm which directly incorporates log-magnitude and phase of the measured electric field data. By doing so, measured phase variation can be unwrapped and distributed over more than one Rieman sheet in the complex plane. Simulation studies and microwave imaging experiments demonstrate that significant image quality enhancements occur with this approach for large high-contrast objects. Simple strategies for visualizing and unwrapping phase values as a function of the transmitter and receiver positions within our microwave imaging array are described. Metrics of the degree of phase variation expressed in terms of the amount and extent of phase wrapping are defined and found to be figures-of-merit which estimate when it is critical to deploy the new image reconstruction approach. In these cases, the new algorithm recovers high-quality images without resorting to the use of a priori information on object contrast and/or size as previously required.     

Interactive image segmentation using Dirichlet process multiple-view learning

Segmenting semantically meaningful whole objects from images is a challenging problem, and it becomes especially so without higher level common sense reasoning. In this paper, we present an interactive segmentation framework that integrates image appearance and boundary constraints in a principled way to address this problem. In particular, we assume that small sets of pixels, which are referred to as seed pixels, are labeled as the object and background. The seed pixels are used to estimate the labels of the unlabeled pixels using Dirichlet process multiple-view learning, which leverages 1) multiple-view learning that integrates appearance and boundary constraints and 2) Dirichlet process mixture-based nonlinear classification that simultaneously models image features and discriminates between the object and background classes. With the proposed learning and inference algorithms, our segmentation framework is experimentally shown to produce both quantitatively and qualitatively promising results on a standard dataset of images. In particular, our proposed framework is able to segment whole objects from images given insufficient seeds.     

Counting Objects by Diffused Index: Geometry-free and training-free approach

Counting objects is a fundamental but challenging problem. In this paper, we propose diffusion-based, geometry-free, and learning-free methodologies to count the number of objects in images. The main idea is to represent each object by a unique index value regardless of its intensity or size, and to simply count the number of index values. First, we place different vectors, refer to as seed vectors, uniformly throughout the mask image. The mask image has boundary information of the objects to be counted. Secondly, the seeds are diffused using an edge-weighted harmonic variational optimization model within each object. We propose an efficient algorithm based on an operator splitting approach and alternating direction minimization method, and theoretical analysis of this algorithm is given. An optimal solution of the model is obtained when the distributed seeds are completely diffused such that there is a unique intensity within each object, which we refer to as an index. For computational efficiency, we stop the diffusion process before a full convergence, and propose to cluster these diffused index values. We refer to this approach as Counting Objects by Diffused Index (CODI). We explore scalar and multi-dimensional seed vectors. For Scalar seeds, we use Gaussian fitting in histogram to count, while for vector seeds, we exploit a high-dimensional clustering method for the final step of counting via clustering. The proposed method is flexible even if the boundary of the object is not clear nor fully enclosed. We present counting results in various applications such as biological cells, agriculture, concert crowd, and transportation. Some comparisons with existing methods are presented.     

Extracting salient region for pornographic image detection

Content-based pornographic image detection, in which region-of-interest (ROI) plays an important role, is effective to filter pornography. Traditionally, skin-color regions are extracted as ROI. However, skin-color regions are always larger than the subareas containing pornographic parts, and the approach is difficult to differentiate between human skins and other objects with the skin-colors. In this paper, a novel approach of extracting salient region is presented for pornographic image detection. At first, a novel saliency map model is constructed. Then it is integrated with a skin-color model and a face detection model to capture ROI in pornographic images. Next, a ROI-based codebook algorithm is proposed to enhance the representative power of visual-words. Taking into account both the speed and the accuracy, we fuse speed up robust features (SURF) with color moments (CM). Experimental results show that the precision of our ROI extraction method averagely achieves 91.33%, more precisely than that of using the skin-color model alone. Besides, the comparison with the state-of-the-art methods of pornographic image detection shows that our approach is able to remarkably improve the performance.     

Region-based image coding using polynomial intensity functions

The vast majority of coded images are real-world images. These images consist of distinct objects within a scene, where each object has its own reflective, textural and lighting characteristics. Region-based image coding encodes these images by partitioning the scene into objects, and then describing each object's characteristics using a set of parameters. The paper uses orthonormal polynomial functions to describe the lighting and reflective characteristics of each object. The coefficients of these polynomials are coded with linear quantisers that have their decision boundaries spaced according to rate-distortion considerations. The textural component of each object is coded using vector quantisation of the autocorrelation coefficients of the residual. The partitioning of the image into distinct objects is achieved with a segmentation algorithm which attempts to maximise the rate-distortion performance of the encoding procedure as a whole. In doing so, the segmentation algorithm partitions the image into distinct objects as well as providing estimates for the optimal bit allocations among the polynomial coefficients. Results generated by this method show reconstructions with quality superior to other region-based methods, both objectively and subjectively     

基于深度学习的图像显著对象检测

显著区域检测可应用在对象识别、图像分割、视 频/图像压缩中,是计算机视觉领域的重要研究主题。然而,基于不 同视觉显著特征的显著区域检测法常常不能准确地探测出显著对象且计算费时。近来,卷积 神经网络模型在图像分析和处理 领域取得了极大成功。为提高图像显著区域检测性能,本文提出了一种基于监督式生成对抗 网络的图像显著性检测方法。它 利用深度卷积神经网络构建监督式生成对抗网络,经生成器网络与鉴别器网络的不断相互对 抗训练,使卷积网络准确学习到 图像显著区域的特征,进而使生成器输出精确的显著对象分布图。同时,本文将网络自身误 差和生成器输出与真值图间的 L1距离相结合,来定义监督式生成对抗网络的损失函数,提升了显著区域检测精度。在MSRA 10K与ECSSD数据库上的实 验结果表明,本文方法 分别获得了94.19%与96.24%的准确率和93.99%与90.13%的召回率,F -Measure值也高达94.15%与94.76%,优于先 前常用的显著性检测模型。     

Preferential Image Segmentation Using Trees of Shapes

A novel preferential image segmentation method is proposed that performs image segmentation and object recognition using mathematical morphologies. The method preferentially segments objects that have intensities and boundaries similar to those of objects in a database of prior images. A tree of shapes is utilized to represent the content distributions in images, and curve matching is applied to compare the boundaries. The algorithm is invariant to contrast change and similarity transformations of translation, rotation and scale. A performance evaluation of the proposed method using a large image dataset is provided. Experimental results show that the proposed approach is promising for applications such as object segmentation and video tracking with cluttered backgrounds.      

Tracking moving object using Snake’s jump based on image flow

An active contour model, Snake, was developed as a useful segmenting and tracking tool for rigid or non-rigid (i.e. deformable) objects by Kass in 1987. Snake is designed on the basis of Snake energies. Segmenting and tracking can be executed successfully by the process of energy minimization. The ability to contract is an important process for segmenting objects from images, but the contraction forces of Kass’ Snake are dependent on the object’s form. In this research, new contraction energy, independent of the object’s form, is proposed for the better segmentation of objects. Kass’ Snake can be applied to the case of small changes between images because its solutions can be achieved on the basis of variational approach. If a somewhat fast moving object exists in successive images, Kass’ Snake will not operate well because the moving object may have large differences in its position or form, between successive images. Snake’s nodes may fall into the local minima in their motion to the new positions of the target object in next image. When the motion is too large to apply image flow energy to tracking, a jump mode is proposed for solving the problem. The vector used to make Snake’s nodes jump to the new location can be obtained by processing the image flow. The effectiveness of the proposed Snake is confirmed by some simulations.     

基于区域特征融合的RGBD显著目标检测

为了对各类自然场景中的显著目标进行检测,本文提出了一种将图像的深度信息引入区域显著性计算的方法,用于目标检测。首先对图像进行多尺度分割得到若干区域,然后对区域多类特征学习构建回归随机森林,采用监督学习的方法赋予每个区域特征显著值,最后采用最小二乘法对多尺度的显著值融合,得到最终的显著图。实验结果表明,本文算法能较准确地定位RGBD图像库中每幅图的显著目标。     

Original approach for the localisation of objects in images

An original approach is presented for the localisation of objects in an image which approach is neuronal and has two steps. In the first step, a rough localisation is performed by presenting each pixel with its neighbourhood to a neural net which is able to indicate whether this pixel and its neighbourhood are the image of the search object. This first filter does not discriminate for position. From its result, areas which might contain an image of the object can be selected. In the second step, these areas are presented to another neural net which can determine the exact position of the object in each area. This algorithm is applied to the problem of localising faces in images     

Optimal bi-directional seam carving for compressibility-aware image retargeting

A novel compressibility-aware image retargeting method based on seam carving is introduced in this paper. We propose a new significance detection method, with both the edge information and visual saliency taken into consideration. A wall-seam model is constructed to evaluate the image compressibility and assign the right number of seams for each direction. By repeatedly carving out or inserting seams we can retarget the image to a new size while preserving the important content. Finally, our algorithm is completed with the supplement of uniformly scaling, the stretched image is resized to the target size with the least structure distortion brought. Experimental results on images show that those improvements are effective and our approach can preserve image content better compared to several state-of-the-art image retargeting methods.     

A wrapper-based approach to image segmentation and classification.

The traditional processing flow of segmentation followed by classification in computer vision assumes that the segmentation is able to successfully extract the object of interest from the background image. It is extremely difficult to obtain a reliable segmentation without any prior knowledge about the object that is being extracted from the scene. This is further complicated by the lack of any clearly defined metrics for evaluating the quality of segmentation or for comparing segmentation algorithms. We propose a method of segmentation that addresses both of these issues, by using the object classification subsystem as an integral part of the segmentation. This will provide contextual information regarding the objects to be segmented, as well as allow us to use the probability of correct classification as a metric to determine the quality of the segmentation. We view traditional segmentation as a filter operating on the image that is independent of the classifier, much like the filter methods for feature selection. We propose a new paradigm for segmentation and classification that follows the wrapper methods of feature selection. Our method wraps the segmentation and classification together, and uses the classification accuracy as the metric to determine the best segmentation. By using shape as the classification feature, we are able to develop a segmentation algorithm that relaxes the requirement that the object of interest to be segmented must be homogeneous in some low-level image parameter, such as texture, color, or grayscale. This represents an improvement over other segmentation methods that have used classification information only to modify the segmenter parameters, since these algorithms still require an underlying homogeneity in some parameter space. Rather than considering our method as, yet, another segmentation algorithm, we propose that our wrapper method can be considered as an image segmentation framework, within which existing image segmentation algorithms may be executed. We show the performance of our proposed wrapper-based segmenter on real-world and complex images of automotive vehicle occupants for the purpose of recognizing infants on the passenger seat and disabling the vehicle airbag. This is an interesting application for testing the robustness of our approach, due to the complexity of the images, and, consequently, we believe the algorithm will be suitable for many other real-world applications.     

Discovering thematic objects in image collections and videos

Given a collection of images or a short video sequence, we define a thematic object as the key object that frequently appears and is the representative of the visual contents. Successful discovery of the thematic object is helpful for object search and tagging, video summarization and understanding, etc. However, this task is challenging because 1) there lacks a priori knowledge of the thematic objects, such as their shapes, scales, locations, and times of re-occurrences, and 2) the thematic object of interest can be under severe variations in appearances due to viewpoint and lighting condition changes, scale variations, etc. Instead of using a top-down generative model to discover thematic visual patterns, we propose a novel bottom-up approach to gradually prune uncommon local visual primitives and recover the thematic objects. A multilayer candidate pruning procedure is designed to accelerate the image data mining process. Our solution can efficiently locate thematic objects of various sizes and can tolerate large appearance variations of the same thematic object. Experiments on challenging image and video data sets and comparisons with existing methods validate the effectiveness of our method.     

Semi-Automatically Labeling Objects in Images

Labeling objects in images plays a crucial role in many visual learning and recognition applications that need training data, such as image retrieval, object detection and recognition. Manually creating object labels in images is time consuming and, thus, becomes impossible for labeling a large image dataset. In this paper, we present a family of semi-automatic methods based on a graph-based semi-supervised learning algorithm for labeling objects in images. We first present SmartLabel that proposes to label images with reduced human input by iteratively computing the harmonic solutions to minimize a quadratic energy function on the Gaussian fields. SmartLabel tackles the problem of lacking negative data in the learning by embedding relevance feedback after the first iteration, which also leads to one limitation of SmartLabel—needing additional human supervision. To overcome the limitation and enhance SmartLabel, we propose SmartLabel-2 that utilizes a novel scheme to sample negative examples automatically, replace regular patch partitioning in SmartLabel by quadtree partitioning and applies image over-segmentation (superpixels) to extract smooth object contours. Evaluation on six diverse object categories have indicated that SmartLabel-2 can achieve promising results with a small amount of labeled data (e.g., 1%–5% of image size) and obtain close-to-fine extraction of object contours on different kinds of objects.      

VORTEX: Video Retrieval and Tracking from Compressed Multimedia Databases—Multiple Object Tracking from MPEG-2 Bit Stream

Multimedia data are generally stored in compressed form in order to efficiently utilize the available storage facilities. Access to multimedia archives is thus dependent on our ability to browse compressed information. In this paper, a novel approach to multiple object tracking from compressed multimedia databases is presented. This approach is intended to operate in a distributed environment, where users initiate video searches and retrieve relevant video information simultaneously from multiple compressed video archives. The system operates on the compressed video to find and track objects of interest and determine their positions in the image. This enables more complex query formulations in terms of the relative positions of the target objects in the image. The filtering and analysis of motion information (motion vectors) is used to track objects in the video bit stream. Once the search has terminated, the system may decompress and display the query-relevant video sequences upon request.