Coupled grouping and matching for sign and gesture recognition

Matching an image sequence to a model is a core problem in gesture or sign recognition. In this paper, we consider such a matching problem, without requiring a perfect segmentation of the scene. Instead of requiring that low- and mid-level processes produce near-perfect segmentation, we take into account that such processes can only produce uncertain information and use an intermediate grouping module to generate multiple candidates. From the set of low-level image primitives, such as constant color region patches found in each image, a ranked set of salient, overlapping, groups of these primitives are formed, based on low-level cues such as region shape, proximity, or color. These groups corresponds to underlying object parts of interest, such as the hands. The sequence of these frame-wise group hypotheses are then matched to a model by casting it into a minimization problem. We show the coupling of these hypotheses with both non-statistical matching (match to sample-based modeling of signs) and statistical matching (match to HMM models) are possible. Our algorithm not only produces a matching score, but also selects the best group in each image frame, i.e. recognition and final segmentation of the scene are coupled. In addition, there is no need for tracking of features across sequences, which is known to be a hard task. We demonstrate our method using data from sign language recognition and gesture recognition, we compare our results with the ground truth hand groups, and achieved less than 5% performance loss for both two models. We also tested our algorithm on a sports video dataset that has moving background.     

Synergy between Object Recognition and Image Segmentation Using the Expectation-Maximization Algorithm

In this work, we formulate the interaction between image segmentation and object recognition in the framework of the Expectation-Maximization (EM) algorithm. We consider segmentation as the assignment of image observations to object hypotheses and phrase it as the E-step, while the M-step amounts to fitting the object models to the observations. These two tasks are performed iteratively, thereby simultaneously segmenting an image and reconstructing it in terms of objects. We model objects using Active Appearance Models (AAMs) as they capture both shape and appearance variation. During the E-step, the fidelity of the AAM predictions to the image is used to decide about assigning observations to the object. For this, we propose two top-down segmentation algorithms. The first starts with an oversegmentation of the image and then softly assigns image segments to objects, as in the common setting of EM. The second uses curve evolution to minimize a criterion derived from the variational interpretation of EM and introduces AAMs as shape priors. For the M-step, we derive AAM fitting equations that accommodate segmentation information, thereby allowing for the automated treatment of occlusions. Apart from top-down segmentation results, we provide systematic experiments on object detection that validate the merits of our joint segmentation and recognition approach.     

结合相似性拟合与空间约束的图像分割

目的 图像中的目标一般含有很多子类,仅仅利用某个子类的特征无法完整地分割出目标区域。针对这一问题,提出一种结合相似性拟合与空间约束的图像交互式分割方法。方法 首先,通过手工标记的样本组成各个目标的字典,通过相似度量搜寻测试样本与各个目标的字典中最相似的原子建立拟合项;再结合图像的空间约束项,构建图像分割模型;最后利用连续最大流算法求解,快速实现图像分割的目的。结果 通过对比实验,本文方法的速度比基于稀疏表示的分类方法的速度提高约13倍,而与归一化切割(N-Cut),逻辑回归(logistic regression)等方法相比,本文方法能取得更稳定和准确的分割结果。此外,本文方法无需过完备字典,只需要训练样本能体现各个子类的信息即可得到稳定的图像分割结果。结论 本文交互式图像分割方法,通过结合相似性拟合以及空间约束建立分割模型,并由连续最大流算法求解,实现图像的快速准确的分割。实验结果表明,该方法能够胜任较准确地对自然图像进行分割以及目标提取等任务。     

Shape Based Detection and Top-Down Delineation Using Image Segments

We introduce a segmentation-based detection and top-down figure-ground delineation algorithm. Unlike common methods which use appearance for detection, our method relies primarily on the shape of objects as is reflected by their bottom-up segmentation. Our algorithm receives as input an image, along with its bottom-up hierarchical segmentation. The shape of each segment is then described both by its significant boundary sections and by regional, dense orientation information derived from the segment’s shape using the Poisson equation. Our method then examines multiple, overlapping segmentation hypotheses, using their shape and color, in an attempt to find a “coherent whole,” i.e., a collection of segments that consistently vote for an object at a single location in the image. Once an object is detected, we propose a novel pixel-level top-down figure-ground segmentation by “competitive coverage” process to accurately delineate the boundaries of the object. In this process, given a particular detection hypothesis, we let the voting segments compete for interpreting (covering) each of the semantic parts of an object. Incorporating competition in the process allows us to resolve ambiguities that arise when two different regions are matched to the same object part and to discard nearby false regions that participated in the voting process. We provide quantitative and qualitative experimental results on challenging datasets. These experiments demonstrate that our method can accurately detect and segment objects with complex shapes, obtaining results comparable to those of existing state of the art methods. Moreover, our method allows us to simultaneously detect multiple instances of class objects in images and to cope with challenging types of occlusions such as occlusions by a bar of varying size or by another object of the same class, that are difficult to handle with other existing class-specific top-down segmentation methods.     

Accurate Object Recognition with Shape Masks

In this paper we propose an object recognition approach that is based on shape masks—generalizations of segmentation masks. As shape masks carry information about the extent (outline) of objects, they provide a convenient tool to exploit the geometry of objects. We apply our ideas to two common object class recognition tasks—classification and localization. For classification, we extend the orderless bag-of-features image representation. In the proposed setup shape masks can be seen as weak geometrical constraints over bag-of-features. Those constraints can be used to reduce background clutter and help recognition. For localization, we propose a new recognition scheme based on high-dimensional hypothesis clustering. Shape masks allow to go beyond bounding boxes and determine the outline (approximate segmentation) of the object during localization. Furthermore, the method easily learns and detects possible object viewpoints and articulations, which are often well characterized by the object outline. Our experiments reveal that shape masks can improve recognition accuracy of state-of-the-art methods while returning richer recognition answers at the same time. We evaluate the proposed approach on the challenging natural-scene Graz-02 object classes dataset.     

Segmentation-based multi-class semantic object detection

In this paper we study the problem of the detection of semantic objects from known categories in images. Unlike existing techniques which operate at the pixel or at a patch level for recognition, we propose to rely on the categorization of image segments. Recent work has highlighted that image segments provide a sound support for visual object class recognition. In this work, we use image segments as primitives to extract robust features and train detection models for a predefined set of categories. Several segmentation algorithms are benchmarked and their performances for segment recognition are compared. We then propose two methods for enhancing the segments classification, one based on the fusion of the classification results obtained with the different segmentations, the other one based on the optimization of the global labelling by correcting local ambiguities between neighbor segments. We use as a benchmark the Microsoft MSRC-21 image database and show that our method competes with the current state-of-the-art.     

Multiscale Active Contours

We propose a new multiscale image segmentation model, based on the active contour/snake model and the Polyakov action. The concept of scale, general issue in physics and signal processing, is introduced in the active contour model, which is a well-known image segmentation model that consists of evolving a contour in images toward the boundaries of objects. The Polyakov action, introduced in image processing by Sochen-Kimmel-Malladi in Sochen et al. (1998), provides an efficient mathematical framework to define a multiscale segmentation model because it generalizes the concept of harmonic maps embedded in higher-dimensional Riemannian manifolds such as multiscale images. Our multiscale segmentation model, unlike classical multiscale segmentations which work scale by scale to speed up the segmentation process, uses all scales simultaneously, i.e. the whole scale space, to introduce the geometry of multiscale images in the segmentation process. The extracted multiscale structures will be useful to efficiently improve the robustness and the performance of standard shape analysis techniques such as shape recognition and shape registration. Another advantage of our method is to use not only the Gaussian scale space but also many other multiscale spaces such as the Perona-Malik scale space, the curvature scale space or the Beltrami scale space. Finally, this multiscale segmentation technique is coupled with a multiscale edge detecting function based on the gradient vector flow model, which is able to extract convex and concave object boundaries independent of the initial condition. We apply our multiscale segmentation model on a synthetic image and a medical image.     

Category Level Object Segmentation by Combining Bag-of-Words Models with Dirichlet Processes and Random Fields

This paper addresses the problem of accurately segmenting instances of object classes in images without any human interaction. Our model combines a bag-of-words recognition component with spatial regularization based on a random field and a Dirichlet process mixture. Bag-of-words models successfully predict the presence of an object within an image; however, they can not accurately locate object boundaries. Random Fields take into account the spatial layout of images and provide local spatial regularization. Yet, as they use local coupling between image labels, they fail to capture larger scale structures needed for object recognition. These components are combined with a Dirichlet process mixture. It models images as a composition of regions, each representing a single object instance. Gibbs sampling is used for parameter estimations and object segmentation.     

多特征组合的深度图像分割算法

深度图像直接反映景物表面的三维几何信息,且不受光照、阴影等因素的影响,对深度图像处理、识别、理解是目前计算机视觉领域研究的热点和重点之一。针对深度图像信息单一且噪声较大的特点,提出一种基于组合特征的阈值分割算法,实现对深度图像数据的有效分割。算法首先通过梯度特征对图像进行Otsu阈值分割;在此基础上,分别在不同分割区域内利用深度特征进行Otsu多阈值分割,得到候选目标;然后,在空域上利用像素的位置特征对候选目标进行分割、合并与去噪,最终得到图像分割的结果。实验结果表明,该方法能有效克服深度图像中噪声的影响,得到的分割区域边界准确,分割质量较高,为以后的室内对象识别和场景理解工作奠定了较好的基础。     

结合形态学和假设检验的视频对象分割

视频对象分割是当前图像和视频处理的热点和难点之一。文章首先采用形态算子和改进的watershed算法对图像序列进行空间分割,然后利用F检测算法进行帧间变化检测,将时空分割结果结合起来,得到初始的变化检测模板。通过相应的基于二值形态算子的后处理,得到最终的分割结果。整个过程基本是对灰度图像和二值模板的形态处理,简单易行。实验结果表明该算法可以较好地分离前景和背景,定位和分割视频对象。     

自上而下注意图分割的细粒度图像分类

目的 针对细粒度图像分类中的背景干扰问题,提出一种利用自上而下注意图分割的分类模型。方法 首先,利用卷积神经网络对细粒度图像库进行初分类,得到基本网络模型。再对网络模型进行可视化分析,发现仅有部分图像区域对目标类别有贡献,利用学习好的基本网络计算图像像素对相关类别的空间支持度,生成自上而下注意图,检测图像中的关键区域。再用注意图初始化GraphCut算法,分割出关键的目标区域,从而提高图像的判别性。最后,对分割图像提取CNN特征实现细粒度分类。结果 该模型仅使用图像的类别标注信息,在公开的细粒度图像库Cars196和Aircrafts100上进行实验验证,最后得到的平均分类正确率分别为86.74%和84.70%。这一结果表明,在GoogLeNet模型基础上引入注意信息能够进一步提高细粒度图像分类的正确率。结论 基于自上而下注意图的语义分割策略,提高了细粒度图像的分类性能。由于不需要目标窗口和部位的标注信息,所以该模型具有通用性和鲁棒性,适用于显著性目标检测、前景分割和细粒度图像分类应用。     

融合分割先验的多图像目标语义分割

目的 在序列图像或多视角图像的目标分割中，传统的协同分割算法对复杂的多图像分割鲁棒性不强，而现有的深度学习算法在前景和背景存在较大歧义时容易导致目标分割错误和分割不一致。为此，提出一种基于深度特征的融合分割先验的多图像分割算法。方法 首先，为了使模型更好地学习复杂场景下多视角图像的细节特征，通过融合浅层网络高分辨率的细节特征来改进PSPNet-50网络模型，减小随着网络的加深导致空间信息的丢失对分割边缘细节的影响。然后通过交互分割算法获取一至两幅图像的分割先验，将少量分割先验融合到新的模型中，通过网络的再学习来解决前景/背景的分割歧义以及多图像的分割一致性。最后通过构建全连接条件随机场模型，将深度卷积神经网络的识别能力和全连接条件随机场优化的定位精度耦合在一起，更好地处理边界定位问题。结果 本文采用公共数据集的多图像集进行了分割测试。实验结果表明本文算法不但可以更好地分割出经过大量数据预训练过的目标类，而且对于没有预训练过的目标类，也能有效避免歧义的区域分割。本文算法不论是对前景与背景区别明显的较简单图像集，还是对前景与背景颜色相似的较复杂图像集，平均像素准确度（PA）和交并比（IOU）均大于95%。结论 本文算法对各种场景的多图像分割都具有较强的鲁棒性，同时通过融入少量先验，使模型更有效地区分目标与背景，获得了分割目标的一致性。     

Stel Component Analysis: Joint Segmentation, Modeling and Recognition of Objects Classes

Models that captures the common structure of an object class have appeared few years ago in the literature (Jojic and Caspi in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 212?C219, 2004; Winn and Jojic in Proceedings of International Conference on Computer Vision (ICCV), pp. 756?C763, 2005); they are often referred as ??stel models.?? Their main characteristic is to segment objects in clear, often semantic, parts as a consequence of the modeling constraint which forces the regions belonging to a single segment to have a tight distribution over local measurements, such as color or texture. This self-similarity within a region in a single image is typical of many meaningful image parts, even when across different images of similar objects, the corresponding parts may not have similar local measurements. Moreover, the segmentation itself is expected to be consistent within a class, although still flexible. These models have been applied mostly to segmentation scenarios. In this paper, we extent those ideas (1) proposing to capture correlations that exist in structural elements of an image class due to global effects, (2) exploiting the segmentations to capture feature co-occurrences and (3) allowing the use of multiple, eventually sparse, observation of different nature. In this way we obtain richer models more suitable to recognition tasks. We accomplish these requirements using a novel approach we dubbed stel component analysis. Experimental results show the flexibility of the model as it can deal successfully with image/video segmentation and object recognition where, in particular, it can be used as an alternative of, or in conjunction with, bag-of-features and related classifiers, where stel inference provides a meaningful spatial partition of features.     

一种针对路口监控图像的区域分割方法

针对智能交通系统中对交通路口场景理解的需求,提出一种基于线特征先验和凸包损失函数的空间分割网络,目标是对斑马线以及斑马线所围路口区域进行精确检测和分割。利用公安交通管理系统平台采集并标注路口数据集;引入线特征先验,将RGBL图像作为网络输入,为深度学习实例分割提供显著的物体边缘特征以加强深度网络对图像特征学习的针对性;在分割网络中引入SCNN网络结构,构成空间分割网络以增强网络对空间结构的学习;引入凸包二值交叉熵动态损失函数来优化网络的输出精度。实验结果表明,该空间分割网络对斑马线及路口区域的检测正确率和分割完整度和精确度都有了显著的提升。     

基于形态学滤波和分水线算法的目标图像分割

提出了一种基于形态学的目标图像区域划分方法。该方法先利用形态学滤波消除不同尺度的噪声和微小干扰区域对目标图像的影响，再利用改进的分水线算法对目标图像进行区域划分，得到目标图像的基本结构。为了消除传统分水线算法引起的过分割现象，本文还给出了一种新的过分割区域合并算法。该方法能够把复杂的目标图像分割成为一系列反映目标基本结构特征的简单区域．为目标的描述和识别提供了方便。实际图像的处理结果显示这种方法行之有效。     

Devanagari OCR using a recognition driven segmentation framework and stochastic language models

This paper describes a novel recognition driven segmentation methodology for Devanagari Optical Character Recognition. Prior approaches have used sequential rules to segment characters followed by template matching for classification. Our method uses a graph representation to segment characters. This method allows us to segment horizontally or vertically overlapping characters as well as those connected along non-linear boundaries into finer primitive components. The components are then processed by a classifier and the classifier score is used to determine if the components need to be further segmented. Multiple hypotheses are obtained for each composite character by considering all possible combinations of the classifier results for the primitive components. Word recognition is performed by designing a stochastic finite state automaton (SFSA) that takes into account both classifier scores as well as character frequencies. A novel feature of our approach is that we use sub-character primitive components in the classification stage in order to reduce the number of classes whereas we use an n-gram language model based on the linguistic character units for word recognition.     

视觉Transformer识别任务研究综述

Transformer模型在自然语言处理领域取得了很好的效果,同时因其能够更好地连接视觉和语言,也激发了计算机视觉界的极大兴趣。本文总结了视觉Transformer处理多种识别任务的百余种代表性方法,并对比分析了不同任务内的模型表现,在此基础上总结了每类任务模型的优点、不足以及面临的挑战。根据识别粒度的不同,分别着眼于诸如图像分类、视频分类的基于全局识别的方法,以及目标检测、视觉分割的基于局部识别的方法。考虑到现有方法在3种具体识别任务的广泛流行,总结了在人脸识别、动作识别和姿态估计中的方法。同时,也总结了可用于多种视觉任务或领域无关的通用方法的研究现状。基于Transformer的模型实现了许多端到端的方法,并不断追求准确率与计算成本的平衡。全局识别任务下的Transformer模型对补丁序列切分和标记特征表示进行了探索,局部识别任务下的Transformer模型因能够更好地捕获全局信息而取得了较好的表现。在人脸识别和动作识别方面,注意力机制减少了特征表示的误差,可以处理丰富多样的特征。Transformer可以解决姿态估计中特征错位的问题,有利于改善基于回归的方法性能,还减少了三维估计时深度映射所产生的歧义。大量探索表明视觉Transformer在识别任务中的有效性,并且在特征表示或网络结构等方面的改进有利于提升性能。     

Modeling and segmentation of floating foreground and background in videos

We propose an on-line algorithm to segment foreground from background in videos captured by a moving camera. In our algorithm, temporal model propagation and spatial model composition are combined to generate foreground and background models, and likelihood maps are computed based on the models. After that, an energy minimization technique is applied to the likelihood maps for segmentation. In the temporal step, block-wise models are transferred from the previous frame using motion information, and pixel-wise foreground/background likelihoods and labels in the current frame are estimated using the models. In the spatial step, another block-wise foreground/background models are constructed based on the models and labels given by the temporal step, and the corresponding per-pixel likelihoods are also generated. A graph-cut algorithm performs segmentation based on the foreground/background likelihood maps, and the segmentation result is employed to update the motion of each segment in a block; the temporal model propagation and the spatial model composition step are re-evaluated based on the updated motions, by which the iterative procedure is implemented. We tested our framework with various challenging videos involving large camera and object motions, significant background changes and clutters.