基于深度信息的实时手势识别和虚拟书写系统

鉴于无接触体感交互技术在人机交互领域的成功应用,提出了一种基于Kinect深度相机的实时隔空虚拟书写方法。结合颜色和深度数据检测和分割出手掌区域;进一步,通过修改的圆扫描转换算法获得手指的个数,以识别不同的手势指令;根据指尖检测从指尖的运动轨迹分割出独立的字符或汉字运动轨迹,并采用随机森林算法识别该字符或汉字。这种基于深度信息的手势检测和虚拟书写方法可以克服光照和肤色重叠的影响,可靠实时地检测和识别手势和隔空书写的文字,其识别率达到93.25%,识别速度达到25 frame/s。     

Estimation of the fundamental matrix from uncalibrated stereo hand images for 3D hand gesture recognition

All 3D hand models employed for hand gesture recognition so far use kinematic models of the hand. We propose to use computer vision models of the hand, and recover hand gestures using 3D reconstruction techniques. In this paper, we present a new method to estimate the epipolar geometry between two uncalibrated cameras from stereo hand images. We first segmented hand images using the RCE neural network based color segmentation algorithm and extracted edge points of fingers as points of interest, then match them based on the topological features of the hand. The fundamental matrix is estimated using a combination of techniques such as input data normalization, rank-2 constraint, linear criterion, nonlinear criterion as well as M-estimator. This method has been tested with real calibrated and uncalibrated images. The experimental comparison demonstrates the effectiveness and robustness of the method.     

基于手势识别的人机交互发展研究

近年来手势识别技术的快速发展,基于手势识别技术的人机交互应用系统的建立使得人机交互的发展前景广阔.从手形、手势和手形手势的建模出发,介绍了模板匹配、特征提取、神经网络和隐马尔可夫模型4种手势识别的方法,并且综述了基于手势识别技术人机交互的发展,详细介绍了3类人机交互系统：漫游型系统、编辑型系统和操作型系统.     

基于手轮廓的深度图像手势识别方法

提出了一种新的手势识别方法,该方法从深度图像中提取手形轮廓,通过计算手形轮廓与轮廓形心点的距离,使用离散傅里叶变换获得手势的表观特征,引入径向基核的支持向量机识别手势。建立了一个常见的10种手势的数据集,测试获得了97.9%的识别率。     

Multi‐view 3‐D display employing a flat‐panel display with slanted pixel arrangement

Abstract— A flat‐panel display with a slanted subpixel arrangement has been developed for a multi‐view three‐dimensional (3‐D) display. A set of 3M × N subpixels (M × N subpixels for each R, G, and B color) corresponds to one of the cylindrical lenses, which constitutes a lenticular lens, to construct each 3‐D pixel of a multi‐view display that offers M × N views. Subpixels of the same color in each 3‐D pixel have different horizontal positions, and the R, G, and B subpixels are repeated in the horizontal direction. In addition, the ray‐emitting areas of the subpixels within a 3‐D pixel are continuous in the horizontal direction for each color. One of the vertical edges of each subpixel has the same horizontal position as the opposite vertical edge of another subpixel of the same color. Cross‐talk among viewing zones is theoretically zero. This structure is suitable for providing a large number of views. A liquid‐crystal panel having this slanted subpixel arrangement was fabricated to construct a mobile 3‐D display with 16 views and a 3‐D resolution of 256 × 192. A 3‐D pixel is comprised of 12 × 4 subpixels (M = 4 and N = 4). The screen size was 2.57 in.     

Eye and gaze tracking for interactive graphic display

This paper describes a computer vision system based on active IR illumination for real-time gaze tracking for interactive graphic display. Unlike most of the existing gaze tracking techniques, which often require assuming a static head to work well and require a cumbersome calibration process for each person, our gaze tracker can perform robust and accurate gaze estimation without calibration and under rather significant head movement. This is made possible by a new gaze calibration procedure that identifies the mapping from pupil parameters to screen coordinates using generalized regression neural networks (GRNNs). With GRNNs, the mapping does not have to be an analytical function and head movement is explicitly accounted for by the gaze mapping function. Furthermore, the mapping function can generalize to other individuals not used in the training. To further improve the gaze estimation accuracy, we employ a hierarchical classification scheme that deals with the classes that tend to be misclassified. This leads to a improvement in classification error. The angular gaze accuracy is about horizontally and vertically. The effectiveness of our gaze tracker is demonstrated by experiments that involve gaze-contingent interactive graphic display.Received: 21 July 2002, Accepted: 3 February 2004, Published online: 8 June 2004 Correspondence to: Qiang Ji     

Electric‐field‐driven LC lens for 3‐D/2‐D autostereoscopic display

Abstract— The use of an electric‐field‐driven liquid‐crystal (ELC) lens cell for switching between a 3‐D and 2‐D display is proposed. Due to the phase retardation of the non‐uniform LC directors, an ELC lens functions the same as a geometric lens. The parameters of an ELC for 3‐D applications are optimized through the simulation of the electrode configuration and voltage levels. A prototype was made where the ELC lens is placed in front of a liquid‐crystal display (LCD) 15 in. on the diagonal with a 99‐μm subpixel pitch. Under zero voltage, the ELC lens is a transparent medium and the users can see a clear 2‐D image. In 3‐D mode, the ELC lens array performs the same as a cylindrical lens array to the incident vertical polarization under suitable driving voltages. Placing a half‐wave plate between the LCD and ELC lens is proposed to change the polarization of the LCD to be parallel with the polarization lens direction of the ELC lens. The measurement of the horizontal luminance profile, performance of the ELC lens, and feasibility for 3‐D/2‐D switching was verified. The fabrication process for the ELC lens is compatible with the current LCD production process and enables the accurate control of the lens pitch of the ELC lens.     

用于手势识别的时空融合网络以及虚拟签名系统

由于新型冠状病毒的流行,非接触式个人签名可以在一定程度上降低感染的风险,其将在人们日常的生活中发挥重要作用。因此,提出了一种简单而有效的时空融合网络来实现基于骨架的动态手势识别,并以此为基础开发了一款虚拟签名系统。时空融合网络主要由基于注意力机制的时空融合模块构成,其核心思想是以增量的方式同步实现时空特征的提取与融合。该网络采用不同编码的时空特征作为输入,并在实际应用中采用双滑动窗口机制来进行后处理,从而确保结果更加的稳定与鲁棒。在2个基准数据集上的大量对比实验表明,该方法优于最先进的单流网络方法。另外,虚拟签名系统在一个普通的RGB相机下表现优异,不仅大大降低了交互系统的复杂性,还提供了一种更为便捷、安全的个人签名方式。     

MUTED: Multi‐user 3‐D display

Abstract— Research described in this paper encompasses the design and building of glasses‐free (autostereoscopic) displays that utilize a direct‐view liquid‐crystal display whose backlight is provided by a projector and novel steering optics. This is controlled by the output of a multi‐user head‐position tracker. As the displays employ spatial multiplexing on a liquid‐crystal‐display screen, they are inherently 2‐D/3‐D switchable with 2‐D being achieved by simply displaying the same image in the left and right channels. Two prototypes are described in this paper; one incorporating a holographic projector and the other a conventional LCOS projector. The LCOS projector version addresses the limitations of brightness, cross‐talk, banding in the images, and laser stability that occur in the holographic projector version. The future development is considered and a comparison between the prototypes and with other 3‐D displays is given.     

结合手指检测和HOG特征的分层静态手势识别

目的 基于手势的交互方式在人机交互中发挥着越来越重要的作用,手势识别是大多数手势交互系统的核心技术.当手势种类较多时,目前已有的大多数手势识别方法往往无法获得足够高的识别率.为此,提出了一种结合手指检测和梯度方向直方图(HOG)特征的分层静态手势识别方法.方法 提出一种基于形态学操作的手指检测算法作为手势识别方法的基础.首先由肤色模型从输入图像中提取出手部区域,然后利用手指检测算法识别出手势包含的手指个数,并根据手指个数从事先训练好的支持向量机分类器集合中选取一个,最后提取手部区域的HOG特征,并利用选择好的分类器完成识别任务.结果 对25种常用手势进行了识别实验,将本文方法与单独使用HOG特征的方法进行对比.本文方法可以将传统HOG方法的识别率提高20%左右.结论 基于手指个数的分层识别策略可以有效地解决传统单层识别方法在手势种类较多时识别率不高的问题.在手部区域能被成功检测的情况下,提出的结合手指检测和HOG特征的方法可以取得较理想的手势识别结果,且能达到实时性要求.     

Feature fusion for 3D hand gesture recognition by learning a shared hidden space

Hand gesture recognition has been intensively applied in various human-computer interaction (HCI) systems. Different hand gesture recognition methods were developed based on particular features, e.g., gesture trajectories and acceleration signals. However, it has been noticed that the limitation of either features can lead to flaws of a HCI system. In this paper, to overcome the limitations but combine the merits of both features, we propose a novel feature fusion approach for 3D hand gesture recognition. In our approach, gesture trajectories are represented by the intersection numbers with randomly generated line segments on their 2D principal planes, acceleration signals are represented by the coefficients of discrete cosine transformation (DCT). Then, a hidden space shared by the two features is learned by using penalized maximum likelihood estimation (MLE). An iterative algorithm, composed of two steps per iteration, is derived to for this penalized MLE, in which the first step is to solve a standard least square problem and the second step is to solve a Sylvester equation. We tested our hand gesture recognition approach on different hand gesture sets. Results confirm the effectiveness of the feature fusion method.     

基于双目视觉的人手定位与手势识别系统研究

提出了一种新的人手特征点提取方法,该方法将人手的质心作为匹配点,根据双目视觉定位数学模型计算目标位置信息,同时通过图像分割获取人手轮廓,利用轮廓凸包点特征来识别不同手势.在此基础上,研究设计了一种光学人手定位与手势识别系统,该系统在实时定位空间人手三维位置的同时,能够识别出相应的手势,可将其作为虚拟手的驱动接口,实现对虚拟物体的抓取、移动和释放操作.     

3‐D crosstalk and luminance uniformity from angular luminance profiles of multiview autostereoscopic 3‐D displays

Abstract— Autostereoscopic 3‐D display technologies enable a more immersive media experience by adding real depth to the visual content. However, the method used for the creation of a sensation of depth or stereo illusion contains several display design and content‐related issues that need to be carefully considered to maintain sufficient image quality. Conventionally, methods used for 3‐D image‐quality evaluations have been based on subjective testing. Optical measurements, in addition to subjective testing, can be used as an efficient tool for 3‐D display characterization. Objective characterization methods for autostereoscopic displays have been developed. How parameters affecting stereo image quality can be defined and measured, and how their effect on the stereo image quality can be evaluated have been investigated. Developed characterization methods are based on empirically gathered data. In this paper, previously presented methodology for two‐view displays is extended to cover autostereoscopic multiview displays. A distinction between displays where the change in content occurs in clear steps when the user moves in front of the display, and displays where the apparent movement of the objects is more continuous as a function of the head movement is made. Definitions for 3‐D luminance and luminance uniformity, which are equally important, as well as 3‐D crosstalk, which is the dominant factor in the evaluations of the perceived 3‐D image quality, is focused upon.     

Design conditions for attractive reality in mobile‐type 3‐D display

Abstract— An autostereoscopic 3‐D display suitable for the mobile environment is prototyped and evaluated. First, the required conditions for a 3‐D display in a mobile environment are considered, and the three major requirements are clarified: small size, viewing‐position flexibility, and application support. An application of a mobile‐type 3‐D display should be different from that of a large‐sized 3‐D display because a mobile‐type 3‐D display cannot realize the feeling of immersion while large‐sized 3‐D displays can realize it easily. From this assumption, it is considered that it is important to realize the feeling to handle a 3‐D image. Three types of 3‐D displays are developed to satisfy these requirements. They are subjectively evaluated to confirm their attractiveness. Results of the tests show that intuitive interaction can increase the reality of the 3‐D image in the sense of unity and also can improve the solidity and depth impression of the 3‐D image.     

Improving resolution and image space in a static volumetric 3‐D display

Abstract— Improvements achieved in image resolution and volume in a volumetric display based on the two‐frequency, two‐step upconversion (TFTS) method are presented. Two digital micromirror devices (DMDs) are utilized to generate fast scanning of the image volume at high resolution. Improvements in resolution and image size over previous implementations are achieved by choosing sodium—ytterbium—fluoride for the imaging crystal instead of the conventional ytterbium—lithium—fluoride composition. Experimentally, images at 532 nm were constructed using 45 slices with each slice rendered at 1024 × 768 resolution, resulting in almost 35 million voxels. The resulting system has the potential to achieve a resolution beyond a targeted 800 million voxels without viewpoint obstruction and with expandability to three‐color imagery.     

Development of a 42‐in. 2‐D/3‐D switchable display using multi‐view technology for public‐information‐display applications

Abstract— A 42‐in. 2‐D/3‐D switchable display operating in a parallax‐barrier‐type system consisting of liquid‐crystal displays (LCDs) has been developed. The system displays 2‐D images in full resolution, without any degradation to the original 2‐D images, and 3‐D autostereoscopic images with resolutions higher than SVGA with wide viewing zones electrically controlled by the parallax‐barrier system. The system is intended for use in public‐information displays (PIDs), a booming field, and as displays for gaming, medical, and simulation applications.     

Static‐volumetric display using rotational‐slicing approach for rendering 3‐D images

Abstract— This work is related to static volumetric crystals which scintillate light when two laser beams are intersected within the crystal. The geometry in this crystal is optimized for linear slices. Most volumetric displays are based on rotational surfaces, which generate the images, while the projected images are sliced in a rotational sweep mode. To date, the majority of 3‐D graphic engines based on static‐volume displays have not been fully developed. To use an advanced 3‐D graphic engine designed for a swept‐volume display (SVD) with a static‐volume display, the display must emulate the operation of a SVD based on a rotational‐slicing approach. The CSpace® 3‐D display has the capability to render 3‐D images using the rotational‐slicing approach. This paper presents the development of a rotational‐slicing approach designed to emulate the operation of a SVD within the image volume of a static‐volume display. The display software has been modified to divide the 3‐D image into 46 slices, each passing through the image center and rotated at a fixed angle from the previous slice. Reconstructed 3‐D images were demonstrated using a rotational‐slicing approach. Suggestions are provided for future implementations that could aid in the elimination of elongations and distortions, which occur within specified slices.     

Optical characterization of autostereoscopic 3‐D displays

Abstract— Display‐measurement methods different from conventional 2‐D display measurements are needed for verifying the optical characteristics of autostereoscopic (3‐D) displays and for comparing different 3‐D display technologies. Industry is lacking standardized measurement methods, and the reported results can not always be compared. The selected set of characteristics discussed in this paper and partly defining the quality of the 3‐D experience are crosstalk, viewing freedom, and optimum viewing distance. Also, more conventional display characteristics such as luminance are discussed, since the definitions for these characteristics in 3‐D mode usually differ from those used for the 2D displays. We have investigated how these chosen 3‐D display characteristics can be objectively measured from transmissive two‐view and multiview 3‐D displays. The scope of this article is to generally define those basic characteristics as well as the different measurement methods. Most of the 3‐D characteristics can be derived from the luminance and colors versus the viewing angle. Either a conoscopic or a goniometric measurement system can be used, as long as the angular and stray‐light properties are suitable and known. The characteristics and methods are currently discussed in the display‐quality standardization forums.     

Laser‐based multi‐user 3‐D display

Abstract— The development of a multi‐user stereoscopic display that does not require the use of special glasses (autostereoscopic), and that enables a large degree of freedom of viewer movement and requires only the minimum amount of information (a stereo pair) for the displays described. The optics comprise an RGB holographic laser projector that is controlled by the output of a multi‐target head‐position head tracker, an optical assembly that converts the projector output into steerable exit pupils, and a screen assembly comprising a single liquid‐crystal display (LCD) and image multiplexing screen. A stereo image pair is produced on the LCD by simultaneously displaying left and right images on alternate rows of pixels. Novel steering optics that replace the conventional backlight are used to direct viewing regions, referred to as exit pupils, to the appropriate viewers' eyes. The results obtained from the first version of the display, where the illumination source consists of several thousand white LEDs, are given and the current status of the latest prototype being constructed on the basis of these results is described. The work indicates that a laser‐based head‐tracking display can provide the basis for the next generation of 3‐D display.     

Dual‐lenticular‐lens‐based 2‐D/3‐D convertible autostereoscopic display

Abstract— A 2‐D/3‐D convertible display using two lenticular lenses has been developed. It shows 2‐D pictures in full resolution and 3‐D autostereoscopic pictures in half resolution by moving one lens relative to the other. The lens assembly consists of thin metal frames, two lenticular lenses, and two shape‐memory‐alloy (SMA) wires used as actuators. While this assembly is applicable to flat‐panel displays of any kind, its simple structure and low power consumption make it best suited to mobile terminals, such as PDAs and mobile phones. Here, we describe its structure and present evaluation results.