An effective graph and depth layer based RGB-D image foreground object extraction method

Computational Visual Media -     

Advanced background modeling with RGB-D sensors through classifiers combination and inter-frame foreground prediction

An innovative background modeling technique that is able to accurately segment foreground regions in RGB-D imagery (RGB plus depth) has been presented in this paper. The technique is based on a Bayesian framework that efficiently fuses different sources of information to segment the foreground. In particular, the final segmentation is obtained by considering a prediction of the foreground regions, carried out by a novel Bayesian Network with a depth-based dynamic model, and, by considering two independent depth and color-based mixture of Gaussians background models. The efficient Bayesian combination of all these data reduces the noise and uncertainties introduced by the color and depth features and the corresponding models. As a result, more compact segmentations, and refined foreground object silhouettes are obtained. Experimental results with different databases suggest that the proposed technique outperforms existing state-of-the-art algorithms.     

Text line extraction in graphical documents using background and foreground information

In graphical documents (e.g., maps, engineering drawings), artistic documents etc., the text lines are annotated in multiple orientations or curvilinear way to illustrate different locations or symbols. For the optical character recognition of such documents, individual text lines from the documents need to be extracted. In this paper, we propose a novel method to segment such text lines and the method is based on the foreground and background information of the text components. To effectively utilize the background information, a water reservoir concept is used here. In the proposed scheme, at first, individual components are detected and grouped into character clusters in a hierarchical way using size and positional information. Next, the clusters are extended in two extreme sides to determine potential candidate regions. Finally, with the help of these candidate regions, individual lines are extracted. The experimental results are presented on different datasets of graphical documents, camera-based warped documents, noisy images containing seals, etc. The results demonstrate that our approach is robust and invariant to size and orientation of the text lines present in the document.     

Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors

In contrast to speech recognition, whose speech features have been extensively explored in the research literature, feature extraction in Sign Language Recognition (SLR) is still a very challenging problem. In this paper we present a methodology for feature extraction in Brazilian Sign Language (BSL, or LIBRAS in Portuguese) that explores the phonological structure of the language and relies on RGB-D sensor for obtaining intensity, position and depth data. From the RGB-D images we obtain seven vision-based features. Each feature is related to one, two or three structural elements in BSL. We investigate this relation between extracted features and structural elements based on shape, movement and position of the hands. Finally we employ Support Vector Machines (SVM) to classify signs based on these features and linguistic elements. The experiments show that the attributes of these elements can be successfully recognized in terms of the features obtained from the RGB-D images, with accuracy results individually above 80% on average. The proposed feature extraction methodology and the decomposition of the signs into their phonological structure is a promising method to help expert systems designed for SLR.     

Efficient and secure key extraction using channel state information

Generating keys and keeping them secret are critical in secure communications. Due to the “open air” nature, key distribution is more susceptible to attacks in wireless communications. An ingenious solution is to generate secret keys for two communicating parties separately without the need of key exchange or distribution, and regenerate them on needs. Recently, it is promising to extract keys by measuring the random variation in wireless channels, e.g., RSS. In this paper, we propose an efficient secret key extraction protocol with decorrelating compressive (SKEDC). It establishes common cryptographic keys for two communicating parties in wireless networks via real-time measurement on channel state information (CSI). It outperforms RSS-based key generation approaches in terms of multiple subcarriers measurement, perfect symmetry in channel for key coincidence, rapid decorrelation with distance, and high sensitivity towards environments changes. In the SKEDC design, we also propose effective mechanisms, such as the adaptive key stream generation, to fully exploit the excellent properties of CSI and eliminate the correlation among the subcarriers. We implement SKEDC on off-the-shelf 802.11n devices and evaluate its performance via extensive experiments. The results demonstrate that SKEDC achieves more than \(3\times \) throughput gain in the key generation from the state-of-the-art RSS-based approaches.     

Efficient extraction of clustering-based feature signatures using GPU architectures

Similarity search and content-based retrieval have become widely used in multimedia database systems that often manage huge data collections. Unfortunately, many effective content-based similarity models cannot be fully utilized for larger datasets, as they are computationally demanding and require massive parallel processing for both feature extraction and query evaluation tasks. In this work, we address the performance issues of effective similarity models based on feature signatures, where we focus on fast feature extraction from image thumbnails using affordable hardware. More specifically, we propose a multi-GPU implementation that increases the extraction speed by two orders of magnitude with respect to a single-threaded CPU implementation. Since the extraction algorithm is not directly parallelizable, we propose a modification of the algorithm embracing the SIMT execution model. We have experimentally verified that our GPU extractor can be successfully used to index large image datasets comprising millions of images. In order to obtain optimal extraction parameters, we employed the GPU extractor in an extensive empirical investigation of the parameter space. The experimental results are discussed from the perspectives of both performance and similarity precision.     

Object recognition in noisy RGB-D data using GNG

Object recognition in 3D scenes is a research field in which there is intense activity guided by the problems related to the use of 3D point clouds. Some of these problems are influenced by the presence of noise in the cloud that reduces the effectiveness of a recognition process. This work proposes a method for dealing with the noise present in point clouds by applying the growing neural gas (GNG) network filtering algorithm. This method is able to represent the input data with the desired number of neurons while preserving the topology of the input space. The GNG obtained results which were compared with a Voxel grid filter to determine the efficacy of our approach. Moreover, since a stage of the recognition process includes the detection of keypoints in a cloud, we evaluated different keypoint detectors to determine which one produces the best results in the selected pipeline. Experiments show how the GNG method yields better recognition results than other filtering algorithms when noise is present.     

Efficient relation extraction method based on spatial feature using ELM

Entity relation extraction can be applied in the automatic question answering system, digital library and many other fields. However, the previous works on this topic mainly focused on the features from a sentence itself in the data sets, without considering the links between sentences in the corpus. In this paper, we propose a concept model and obtain a new effective spatial feature based on this concept model. The added feature makes our feature space concerning not only the inherent information of the sentence itself, but also the semantic information connection between sentences. At last, we use ELM as the training classifier in entity relation extraction. The experiment result shows that the precision and recall of the relation extraction both have a significant increase, by using the new feature. Also, the use of ELM significantly reduces the time of relation extraction. It has a better performance than the traditional method based on SVM.     

Efficient representation and extraction of 2-manifold isosurfaces using kd-trees

In this paper, we propose the utilization of a kd-tree based hierarchy as an implicit object representation. Compared to an octree, the kd-tree based hierarchy is superior in terms of adaptation to the object surface. In consequence, we obtain considerably more compact implicit representations especially in case of thin object structures. We describe a new isosurface extraction algorithm for this kind of implicit representation. In contrast to related algorithms for octrees, it generates 2-manifold meshes even for kd-trees with cells containing multiple surface components. The algorithm retains all the good properties of the Dual Contouring approach by Ju et al. [ACM Trans. Graphics 21 (2002) 339–346] like feature preservation, computational efficiency, etc. In addition, we present a simplification framework for the surfaces represented by the kd-tree based on quadric error metrics. We adapt this framework to quantify the influence of topological changes, thereby allowing controlled topological simplification of the object. The advantages of the new algorithm are demonstrated by several examples.     

Realistic surface geometry reconstruction using a hand-held RGB-D camera

In this paper, we have proposed a novel approach for the reconstruction of real object/scene with realistic surface geometry using a hand-held, low-cost, RGB-D camera. To achieve accurate reconstruction, the most important issues to consider are the quality of the geometry information provided and the global alignment method between frames. In our approach, new surface geometry refinement is used to recover finer scale surface geometry from depth data by utilizing high-quality RGB images. In addition, a weighted multi-scale iterative closest point method is exploited to align each scan to the global model accurately. We show the effectiveness of the proposed surface geometry refinement method by comparing it with other depth refinement methods. We also show both the qualitative and quantitative results of reconstructed models by comparing it with other reconstruction methods.     

基于RGB-D单目视觉的室内场景三维重建

针对室内环境单目视觉的室内场景三维重建速度慢的问题,采用华硕Xtion单目视觉传感器获取的室内场景彩色图像和深度图像进行快速三维重建。在图像特征提取上使用ORB特征检测算法,并对比了几种经典特征检测算法在图像匹配中的效率,在图像匹配融合中采用Ransac算法和ICP算法进行点云融合。实现了一种室内简单、小规模的静态环境快速三维重建方法,通过实验验证了该方法有较好的精确性、鲁棒性、实时性和灵活性。     

Robust facial expression recognition using RGB-D images and multichannel features

Multimedia Tools and Applications - Traditional and classical methods of facial expression recognition are mainly based on intensity image and are prone to be disturbed by illumination, poses, and...     

Robust foreground detection in video using pixel layers

A framework for robust foreground detection that works under difficult conditions such as dynamic background and moderately moving camera is presented in this paper. The proposed method includes two main components: coarse scene representation as the union of pixel layers, and foreground detection in video by propagating these layers using a maximum-likelihood assignment. We first cluster into "layers" those pixels that share similar statistics. The entire scene is then modeled as the union of such non-parametric layer-models. An in-coming pixel is detected as foreground if it does not adhere to these adaptive models of the background. A principled way of computing thresholds is used to achieve robust detection performance with a pre-specified number of false alarms. Correlation between pixels in the spatial vicinity is exploited to deal with camera motion without precise registration or optical flow. The proposed technique adapts to changes in the scene, and allows to automatically convert persistent foreground objects to background and re-convert them to foreground when they become interesting. This simple framework addresses the important problem of robust foreground and unusual region detection, at about 10 frames per second on a standard laptop computer. The presentation of the proposed approach is complemented by results on challenging real data and comparisons with other standard techniques.     

Healthy human sitting posture estimation in RGB-D scenes using object context

Multimedia Tools and Applications - Unhealthy sitting posture leads to cervical spondylosis and other related cumulative trauma disorders (CTDs). Unfortunately, the research on the investigation of...     

Building 3D semantic maps for mobile robots using RGB-D camera

The wide availability of affordable RGB-D sensors changes the landscape of indoor scene analysis. Years of research on simultaneous localization and mapping (SLAM) have made it possible to merge multiple RGB-D images into a single point cloud and provide a 3D model for a complete indoor scene. However, these reconstructed models only have geometry information, not including semantic knowledge. The advancements in robot autonomy and capabilities for carrying out more complex tasks in unstructured environments can be greatly enhanced by endowing environment models with semantic knowledge. Towards this goal, we propose a novel approach to generate 3D semantic maps for an indoor scene. Our approach creates a 3D reconstructed map from a RGB-D image sequence firstly, then we jointly infer the semantic object category and structural class for each point of the global map. 12 object categories (e.g. walls, tables, chairs) and 4 structural classes (ground, structure, furniture and props) are labeled in the global map. In this way, we can totally understand both the object and structure information. In order to get semantic information, we compute semantic segmentation for each RGB-D image and merge the labeling results by a Dense Conditional Random Field. Different from previous techniques, we use temporal information and higher-order cliques to enforce the label consistency for each image labeling result. Our experiments demonstrate that temporal information and higher-order cliques are significant for the semantic mapping procedure and can improve the precision of the semantic mapping results.     

CAD-based Pose Estimation Design for Random Bin Picking using a RGB-D Camera

This paper presents a CAD-based six-degrees-of-freedom (6-DoF) pose estimation design for random bin picking for multiple objects. A virtual camera generates a point cloud database for the objects using their 3D CAD models. To reduce the computational time of 3D pose estimation, a voxel grid filter reduces the number of points for the 3D cloud of the objects. A voting scheme is used for object recognition and to estimate the 6-DoF pose for different objects. An outlier filter filters out badly matching poses so that the robot arm always picks up the upper object in the bin, which increases the success rate. In a computer simulation using a synthetic scene, the average recognition rate is 97.81 % for three different objects with various poses. A series of experiments have been conducted to validate the proposed method using a Kuka robot arm. The average recognition rate for three objects is 92.39 % and the picking success rate is 89.67 %.     

Wound measurement by RGB-D camera

The robot and computer vision community has seen a lot of novelties developed in the past few years as a result of the appearance of cheap RGB-D sensors spearheaded by the Kinect sensor. In this paper, the feasibility of using an RGB-D camera in detecting, segmenting, reconstructing and measuring chronic wounds in 3D is explored. The wound is detected by implementing nearest-neighbor approach on color histograms generated from the image. The proposed wound segmentation procedure extracts the wound contour using visual and geometrical information of the surface. A procedure comparable to KinectFusion is used for the 3D reconstruction of the wound. In order to achieve real-time performance, the whole system is realized in CUDA. The resulting system provides an accurate colored 3D model of the segmented wound and enables the user to determine the volume, area and perimeter of the wound, thereby aiding in the selection of a suitable therapy. The developed system is experimentally evaluated using the Saymour II wound care model by VATA Inc.