Nonstationary color tracking for vision-based human-computer interaction

Skin color offers a strong cue for efficient localization and tracking of human body parts in video sequences for vision-based human-computer interaction. Color-based target localization could be achieved by analyzing segmented skin color regions. However, one of the challenges of color-based target tracking is that color distributions would change in different lighting conditions such that fixed color models would be inadequate to capture nonstationary color distributions over time. Meanwhile, using a fixed skin color model trained by the data of a specific person would probably not work well for other people. Although some work has been done on adaptive color models, this problem still needs further studies. We present our investigation of color-based image segmentation and nonstationary color-based target tracking, by studying two different representations for color distributions. We propose the structure adaptive self-organizing map (SASOM) neural network that serves as a new color model. Our experiments show that such a representation is powerful for efficient image segmentation. Then, we formulate the nonstationary color tracking problem as a model transduction problem, the solution of which offers a way to adapt and transduce color classifiers in nonstationary color distributions. To fulfill model transduction, we propose two algorithms, the SASOM transduction and the discriminant expectation-maximization (EM), based on the SASOM color model and the Gaussian mixture color model, respectively. Our extensive experiments on the task of real-time face/hand localization show that these two algorithms can successfully handle some difficulties in nonstationary color tracking. We also implemented a real-time face/hand localization system based on such algorithms for vision-based human-computer interaction.     

A color-based face tracking algorithm for enhancing interaction with mobile devices

A color-based face tracking algorithm is proposed to be used as a human-computer interaction tool on mobile devices. The solution provides a natural means of interaction enabling a motion parallax effect in applications. The algorithm considers the characteristics of mobile use-constrained computational resources and varying environmental conditions. The solution is based on color comparisons and works on images gathered from the front camera of a device. In addition to color comparisons, the coherency of the facial pixels is considered in the algorithm. Several applications are also demonstrated in this work, which use the face position to determine the viewpoint in a virtual scene, or for browsing large images. The accuracy of the system is tested under different environmental conditions such as lighting and background, and the performance of the system is measured in different types of mobile devices. According to these measurements the system allows for accurate (7% RMS error) face tracking in real time (20–100 fps).     

Physics-based modelling of human skin colour under mixed illuminants

Skin colour is an often used feature in human face and motion tracking. It has the advantages of being orientation and size invariant and it is fast to process. The major disadvantage is that it becomes unreliable if the illumination changes. In this paper, skin colour is modelled based on a reflectance model of the skin, the parameters of the camera and light sources. In particular, the location of the skin colour area in the chromaticity plane is modelled for different and mixed light sources. The model is empirically validated. It has applications in adaptive segmentation of skin colour and in the estimation of the current illumination in camera images containing skin colour.     

基于Lab空间的图像检索算法

探讨Lab颜色空间内基于颜色的图像检索问题。分析已有的基于Lab色度直方图的检索算法的不足,提出改进的基于Lab空间颜色通道的检索算法,该算法分别将a, b颜色通道非均匀量化成22个等级,用其直方图来表征图像的颜色特征,从而保留了两个颜色通道的特性。设计并实现了基于用户感兴趣图像块的相关反馈技术。实验结果表明,改进算法具有良好的检索结果,采用的相关反馈技术提高了检索性能。     

自适应融合颜色和深度信息的人体轮廓跟踪

采用活动轮廓对人体目标建模,提出一 种新的水平集框架下自适应融合RGB-D图像的颜色和深度信息的人体轮廓跟踪方法. 设计了一种基于超像素的局部自适应权重计算方法,自动确定深度信息在水平集演化中的重要性. 基于深度信息的活动轮廓驱动外力包括由边缘生成的梯度向量流和由目标/背景深度模型生成的置信图,基于颜色信息的驱动外力由目标/背景颜色模型生成的置信图,这三种外力通过局部自适应权重融合,驱动活动轮廓向目标的边界演化.为了得到更加精确的目标轮廓和防止误差漂移,基于本文观察到的人体表面在深度图像中的两个特性,提出两个简单但有效的算法对水平集方法得到的结果进行精化调整. 最后,通过实验验证了本文算法的优越性.     

利用基于颜色的自适应形状模型实现手势跟踪*

提出了一种基于颜色的自适应形状模型,并利用该模型实现了图像序列中的实时手势跟踪.跟踪算法基于自适应的颜色模型实现准确的手部轮廓提取,并利用基于二维颜色模型的粒子滤波器实现序列图像中跟踪目标的运动估计.实验结果表明了基于颜色的自适应形状模型对凸形与凹形手部轮廓均能达到准确的手部轮廓提取,并能满足图像序列手势跟踪的实行性要求.     

嵌入卡尔曼预测器的粒子滤波目标跟踪算法*

针对经典的粒子滤波视频目标跟踪算法进行粒子传播采用随机游走的方式,以及传统颜色直方图无法反映目标空间特征的问题,提出了一种改进的基于颜色的粒子滤波目标跟踪算法。该算法在统计目标二阶颜色直方图的基础上,获得粒子的观察概率密度函数,利用卡尔曼滤波确定粒子动态传播模型中的确定性漂移部分,使粒子状态估计值分布更精确地趋向目标的概率分布,大大提高了粒子的利用效率。实验表明,该改进算法的性能优于经典基于单一颜色特征的粒子滤波算法。     

Skin segmentation based on multi pixel color clustering models

This paper presents a reliable color pixel clustering model for skin segmentation under unconstrained scene conditions. The proposed model can overcome sensitivity to variations in lighting conditions and complex backgrounds. Our approach is based on building multi-skin color clustering models using the Hue, Saturation, and Value color space and multi-level segmentation. Skin regions are extracted using four skin color clustering models, namely, the standard-skin, shadow-skin, light-skin, and high-red-skin models. Moreover, skin color correction (skin lighting) at the shadow-skin layer is used to improve the detection rate. The experimental results from a large image data set demonstrate that the proposed clustering models could achieve a true positive rate of 96.5% and a false positive rate of approximately 0.765%. The experimental results show that the color pixel clustering model is more efficient than other approaches.     

Statistical Modeling of Colour Data

In this paper we investigate how best to model naturally arising distributions of colour camera data. It has become standard to model single mode distributions of colour data by ignoring the intensity component and constructing a Gaussian model of the chromaticity. This approach is appealing, because the intensity of data can change arbitrarily due to shadowing and shading, whereas the chromaticity is more robust to these effects. However, it is unclear how best to construct such a model, since there are many domains in which the chromaticity can be represented. Furthermore, the applicability of this kind of model is questionable in all but the most basic lighting environments.We begin with a review of the reflection processes that give rise to distributions of colour data. Several candidate models are then presented; some are from the existing literature and some are novel. Properties of the different models are compared analytically and the models are empirically compared within a region tracking application over two separate sets of data. Results show that chromaticity based models perform well in constrained environments where the physical model upon which they are based applies. It is further found that models based on spherical representations of the chromaticity data provide better performance than those based on more common planar representations, such as the chromaticity plane or the normalised colour space. In less constrained environments, however, such as daylight, chromaticity based models do not perform well, because of the effects of additional illumination components, which violate the physical model upon which they are based.     

Segmentation of Different Skin Colors with Different Lighting Conditions by Combining Graph Cuts Algorithm with Probability Neural Network Classification, and its Application

It is realized that fixed thresholds mostly fail in two circumstances as they only search for a certain range of skin color: (i) any skin-like object may be classified as skin if skin-like colors belong to fixed threshold range; (ii) any true skin for different races may be mistakenly classified as non-skin if that skin colors do not belong to fixed threshold range. In this paper, graph cuts (GC) is first extended to skin color segmentation. Although its result is acceptable, a complex environment with skin-like objects or different skin colors or different lighting conditions often results in a partial success. It is also known that probability neural network (PNN) has the advantage of recognizing different skin colors in cluttered environments. Therefore, many images with skin-like objects or different skin colors or different lighting conditions are segmented by the proposed algorithm (i.e., the combination of GC algorithm and PNN classification with other functions, e.g., morphology filtering, labeling, area constraint). The compared results among GC algorithm, PNN classification, and the proposed algorithm are presented not only to verify the accurate segmentation of these images but also to reduce the computation time. Finally, the application to the classification of hand gestures in complex environment with different lighting conditions further confirms the effectiveness and efficiency of our method.     

YCbCr空间中一种基于贝叶斯判决的肤色检测方法

皮肤颜色是人脸检测、定位、跟踪时的一种十分有效的特征,而且裸露的皮肤区域也是色情图像的最重要特征之一.为了有效地进行图像的皮肤检测,提出了一种新的肤色检测方法.该方法首先通过统计1809 502个肤色像素点和1763682个非肤色像素点,并使用贝叶斯规则来建立肤色分类器;然后考虑亮度对肤色的影响,采用Y-Cb和Y-Cr两个子空间的查询表来建立肤色模型.为了联合使用两个查询表,先采用高斯归一化和线性化方法来将阈值范围调整至[0,1];同时对查询表进行中值滤波处理,以除去离散孤立点.实验表明,与其他3种方法相比,该方法不仅有着较低的漏检率（9.814%）和误检率（3.5%）,而且对于不同光照条件也有较好的检测效果.     

彩色视频序列图像中的人脸跟踪方法

针对彩色视频序列图像的人脸检测，提出了一种基于肤色的人脸跟踪方法。该方法首先在Hsu提出的肤色模型基础上，采样一种自肤色分割算法来提取复杂背景下人脸的肤色特征，与传统的采用固定肤色模型的检测算法相比，该方法具有更好的检测效果；然后，在人脸跟踪过程中采用Condensation滤波跟踪算法，并对算法做了两点改进，即在跟踪过程中采用基于Metropolis算法的重采样方法以及自适应的动态模型，实现了复杂背景下的人脸自由运动的跟踪，并从各种影片中截取了彩色视频序列图像进行了测试实验。实验结果表明，该方法有效地解决了复杂背景下人脸自由运动、光照变化及部分遮挡的问题，且精度较高。     

Video Relighting Using Infrared Illumination

Inappropriate lighting is often responsible for poor quality video. In most offices and homes, lighting is not designed for video conferencing. This can result in unevenly lit faces, distracting shadows, and unnatural colors. We present a method for relighting faces that reduces the effects of uneven lighting and color. Our setup consists of a compact lighting rig and a camera that is both inexpensive and inconspicuous to the user. We use unperceivable infrared (IR) lights to obtain an illumination bases of the scene. Our algorithm computes an optimally weighted combination of IR bases to minimize lighting inconsistencies in foreground areas and reduce the effects of colored monitor light. However, IR relighting alone results in images with an unnatural ghostly appearance, thus a retargeting technique is presented which removes the unnatural IR effects and produces videos that have substantially more balanced intensity and color than the original video.     

DeepGarment : 3D Garment Shape Estimation from a Single Image

3D garment capture is an important component for various applications such as free‐view point video, virtual avatars, online shopping, and virtual cloth fitting. Due to the complexity of the deformations, capturing 3D garment shapes requires controlled and specialized setups. A viable alternative is image‐based garment capture. Capturing 3D garment shapes from a single image, however, is a challenging problem and the current solutions come with assumptions on the lighting, camera calibration, complexity of human or mannequin poses considered, and more importantly a stable physical state for the garment and the underlying human body. In addition, most of the works require manual interaction and exhibit high run‐times. We propose a new technique that overcomes these limitations, making garment shape estimation from an image a practical approach for dynamic garment capture. Starting from synthetic garment shape data generated through physically based simulations from various human bodies in complex poses obtained through Mocap sequences, and rendered under varying camera positions and lighting conditions, our novel method learns a mapping from rendered garment images to the underlying 3D garment model. This is achieved by training Convolutional Neural Networks (CNN‐s) to estimate 3D vertex displacements from a template mesh with a specialized loss function. We illustrate that this technique is able to recover the global shape of dynamic 3D garments from a single image under varying factors such as challenging human poses, self occlusions, various camera poses and lighting conditions, at interactive rates. Improvement is shown if more than one view is integrated. Additionally, we show applications of our method to videos.     

Detection and correction of purple fringing using color desaturation in the xy chromaticity diagram and the gradient information

This paper proposes a method to detect and correct purple fringing, which is one of color artifacts due to characteristics of charge coupled device sensors in a digital camera. The proposed method consists of two steps: detection and correction of purple fringing. In the first step, we detect the purple fringed regions that satisfy specific properties: image regions with large gradient magnitudes and with the chromaticity values within the purple color range set in the CIE xy chromaticity diagram. In the second step, color of the purple fringed regions is corrected in the CIE xy chromaticity diagram by color desaturation of the detected pixels. The proposed method is able to detect purple fringe artifacts more precisely and correct them more naturally than existing methods. It can be used as a post processing in a digital camera.     

Do mixture models in chromaticity space improve skin detection?

This note reports an experiment where a single Gaussian model and several Gaussian mixture models were used to model skin color in the rg chromaticity space. By using training and test databases containing millions of skin pixels, we show that mixture models can improve skin detection, but not always. There is a relevant operating region where no performance gain is observed.     

Real-time automatic license plate recognition for CCTV forensic applications

We propose an efficient real-time automatic license plate recognition (ALPR) framework, particularly designed to work on CCTV video footage obtained from cameras that are not dedicated to the use in ALPR. At present, in license plate detection, tracking and recognition are reasonably well-tackled problems with many successful commercial solutions being available. However, the existing ALPR algorithms are based on the assumption that the input video will be obtained via a dedicated, high-resolution, high-speed camera and is/or supported by a controlled capture environment, with appropriate camera height, focus, exposure/shutter speed and lighting settings. However, typical video forensic applications may require searching for a vehicle having a particular number plate on noisy CCTV video footage obtained via non-dedicated, medium-to-low resolution cameras, working under poor illumination conditions. ALPR in such video content faces severe challenges in license plate localization, tracking and recognition stages. This paper proposes a novel approach for efficient localization of license plates in video sequence and the use of a revised version of an existing technique for tracking and recognition. A special feature of the proposed approach is that it is intelligent enough to automatically adjust for varying camera distances and diverse lighting conditions, a requirement for a video forensic tool that may operate on videos obtained by a diverse set of unspecified, distributed CCTV cameras.     

A local-motion-based probabilistic model for visual tracking

Color-based tracking is prone to failure in situations where visually similar targets are moving in a close proximity or occlude each other. To deal with the ambiguities in the visual information, we propose an additional color-independent visual model based on the target's local motion. This model is calculated from the optical flow induced by the target in consecutive images. By modifying a color-based particle filter to account for the target's local motion, the combined color/local-motion-based tracker is constructed. We compare the combined tracker to a purely color-based tracker on a challenging dataset from hand tracking, surveillance and sports. The experiments show that the proposed local-motion model largely resolves situations when the target is occluded by, or moves in front of, a visually similar object.     

基于彩色图像处理的西红柿品质特征的提取研究

用机器视觉系统判别西红柿品质是否有效 ,关键在于能否真实地提取特征参数 .由于西红柿果实存在新陈代谢且表面非常光滑 ,使机器视觉系统采集图像时受环境温度和光线的影响很大 .同时 ,很难直接从采集到的RGB彩色图像中提取有效的品质特征参数 .为此 ,本论文基于彩色图像处理技术把 RGB彩色图像转换为 L* a* b*模式的彩色图像 ,然后提取西红柿品质特征参数 ,试验结果验证采用这种特征提取法提取的西红柿品质特征参数基本上不受照明强度的影响     

Affine Reconstruction from Translational Motion under Various Autocalibration Constraints

In this paper algorithms for affine reconstruction from translational motion under various auto calibration constraints are presented. A general geometric constraint, expressed using the camera matrices, is derived and this constraint is used in a least square solution to the problem. Necessary and sufficient conditions for critical motions are derived and shown to depend on the knowledge of the intrinsic parameters of the camera. Experiments on simulated data are performed to evaluate the noise sensitivity of the algorithms and the reconstruction quality for motions close to being critical. An experiment is performed on real data to illustrate that the method works in practice.