Human–computer interaction using vision-based hand gesture recognition systems: a survey

Considerable effort has been put toward the development of intelligent and natural interfaces between users and computer systems. In line with this endeavor, several modes of information (e.g., visual, audio, and pen) that are used either individually or in combination have been proposed. The use of gestures to convey information is an important part of human communication. Hand gesture recognition is widely used in many applications, such as in computer games, machinery control (e.g., crane), and thorough mouse replacement. Computer recognition of hand gestures may provide a natural computer interface that allows people to point at or to rotate a computer-aided design model by rotating their hands. Hand gestures can be classified into two categories: static and dynamic. The use of hand gestures as a natural interface serves as a motivating force for research on gesture taxonomy, its representations, and recognition techniques. This paper summarizes the surveys carried out in human--computer interaction (HCI) studies and focuses on different application domains that use hand gestures for efficient interaction. This exploratory survey aims to provide a progress report on static and dynamic hand gesture recognition (i.e., gesture taxonomies, representations, and recognition techniques) in HCI and to identify future directions on this topic.     

基于Hausdorff距离的手势识别

随着先进人机交互技术的提出及发展，手势识别正成为其中一项关键技术，基于视觉的手势识别是当前涉及图象处理，模式识别，计算机视觉等领域的一个比较活跃的课题，由于Hausdorff距离模板匹配的方法具有计算量小，适应性强的特点，因此基于Hausdorff距离，建立了一个手势识别系统，该系统采用边缘特征像素点作为识别特征，并首次利用Hausdorff距离模板匹配的思想，在距离变换空间内，实现了中国手指字母集上的基于单目视觉的30个手指字母的手势识别，为提高系统的鲁棒性，还提出了修正的Hausdorff距离形式，测试集上的平均识别率为96．7％，实验结果表明，基于Hausdorff距离的模板匹配方法用于基于听觉的静态手势识别是可行的。     

Real-time single camera natural user interface engine development

Natural user interfaces (NUIs) provide human computer interaction (HCI) with natural and intuitive operation interfaces, such as using human gestures and voice. We have developed a real-time NUI engine architecture using a web camera as a means of implementing NUI applications. The system captures video via the web camera, implements real-time image processing using graphic processing unit (GPU) programming. This paper describes the architecture of the engine and the real-virtual environment interaction methods, such as foreground segmentation and hand gesture recognition. These methods are implemented using GPU programming in order to realize real-time image processing for HCI. To verify the efficacy of our proposed NUI engine, we utilized it in the development and implementation of several mixed reality games and touch-less operation applications, using the developed NUI engine and the DirectX SDK. Our results confirm that the methods implemented by the engine operate in real time and the interactive operations are intuitive.     

基于Kinect手势识别的应用与研究

为解决当前智能家居系统操作繁琐的问题,同时为获得更简单的控制方式,并增加用户的体验感受,研究了基于Kinect骨骼信息的手势识别技术,并将其融入至智能家居的人机交互系统中。在该系统中,用户可以自定义手势动作或语音实现家居设备的智能控制。使用了一种基于加权动态时间规整的模板匹配手势识别算法。通过Kinect的深度摄像头获取手势深度图像和骨骼图像数据,并采用加权动态时间规整算法进行识别。实验表明使用该算法实现手势识别是可行且有效的,且其最佳识别位置是在Kinect的正前方2～2.5m处,识别准确率达到96%左右。     

Visual interpretation of hand gestures for human-computerinteraction: a review

The use of hand gestures provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). In particular, visual interpretation of hand gestures can help in achieving the ease and naturalness desired for HCI. This has motivated a very active research area concerned with computer vision-based analysis and interpretation of hand gestures. We survey the literature on visual interpretation of hand gestures in the context of its role in HCI. This discussion is organized on the basis of the method used for modeling, analyzing, and recognizing gestures. Important differences in the gesture interpretation approaches arise depending on whether a 3D model of the human hand or an image appearance model of the human hand is used. 3D hand models offer a way of more elaborate modeling of hand gestures but lead to computational hurdles that have not been overcome given the real-time requirements of HCI. Appearance-based models lead to computationally efficient “purposive” approaches that work well under constrained situations but seem to lack the generality desirable for HCI. We also discuss implemented gestural systems as well as other potential applications of vision-based gesture recognition. Although the current progress is encouraging, further theoretical as well as computational advances are needed before gestures can be widely used for HCI. We discuss directions of future research in gesture recognition, including its integration with other natural modes of human-computer interaction     

Concept-based evidential reasoning for multimodal fusion in human–computer interaction

Considerable research has been done on using information from multiple modalities, like hands, facial gestures or speech, for better interaction between humans and computers, and many promising human–computer interfaces (HCI) have been developed in recent years. However, most of the current HCI systems have a few drawbacks: firstly, they are highly dependent on the performance of individual sensors. S econdly, the information fusion process from these sensors tends to ignore the semantic nature of the modalities, which may reinforce or clarify each other over time. Finally, they are not robust enough at representing the imprecise nature of human gestures, since individual gestures are highly ambiguous in themselves. In this paper, we propose an approach for the semantic fusion of different input modalities, based on transferable belief models. We show that this approach allows for a better representation of the ambiguity involved in recognizing gestures. Ambiguity is resolved by combining the beliefs of the individual sensors on the input information, to form new extended concepts, based on a pre-defined domain specific knowledge base, represented by conceptual graphs. We apply this technique to a multimodal system consisting of a hand gesture recognition sensor and a brain computing interface. It is shown that the technique can successfully combine individual gestures obtained from the two sensors, to form meaningful concepts and resolve ambiguity. The advantage of this approach is that it is robust even if one of the sensors is inefficient or has no input. Another important feature is its scalability, wherein more input modalities, like speech or facial gestures, can be easily integrated into the system at minimal cost, to form a comprehensive HCI interface.     

GIS中线元的熵不确定带研究

空间数据的不确定性将直接影响地理信息产品的质量有GIS空间决策的可靠性，现已把它作为一个重要的基础理论问题加以研究，其中线元的位置不确定性是研究的一个热点，针对现有的线元位置不确定性模型的不足，通过引入信息熵理论，首先提出了二维随机点的熵误差椭圆指标与三维随机点的熵误差椭球指标；然后将它们扩展到线元的熵不确定带，实践证明，由于该模型能够根据联合熵唯一确定，且与置信水平的选取无关，因此比较适合作为线元位置不确定性度量的指标。     

A virtual mouse interface with a two-layered Bayesian network

During the last decade, many natural interaction methods between human and computer have been introduced. They were developed for substitutions of keyboard and mouse devices so that they provide convenient interfaces. Recently, many studies on vision based gestural control methods for Human-Computer Interaction (HCI) have been attracted attention because of their convenience and simpleness. Two of the key issues in these kinds of interfaces are robustness and real-time processing. This paper presents a hand gesture based virtual mouse interface and Two-layer Bayesian Network (TBN) for robust hand gesture recognition in real-time. The TBN provides an efficient framework to infer hand postures and gestures not only from information at the current time frame, but also from the preceding and following information, so that it compensates for erroneous postures and its locations under cluttered background environment. Experiments demonstrated that the proposed model recognized hand gestures with a recognition rate of 93.76 % and 85.15 % on simple and cluttered background video data, respectively, and outperformed previous methods: Hidden Markov Model (HMM), Finite State Machine (FSM).     

智能交互的物体识别增量学习技术综述

智能交互系统是研究人与计算机之间进行交流与通信,使计算机能够在最大程度上完成交互者的某个指令的一个领域。其发展的目标是实现人机交互的自主性、安全性和友好性。增量学习是实现这个发展目标的一个途径。本文对智能交互系统的任务、背景和获取信息来源进行简要介绍,主要对增量学习领域的已有工作进行综述。增量学习是指一个学习系统能不断地从新样本中学习新的知识,非常类似于人类自身的学习模式。它使智能交互系统拥有自我学习,提高交互体验的能力。文中对主要的增量学习算法的基本原理和特点进行了阐述,分析各自的优点和不足,并对进一步的研究方向进行展望。     

智能人机交互中第一视角手势表达的一次性学习分类识别

在智能人机交互中, 以交互人的视角为第一视角的手势表达发挥着重要作用, 而面向第一视角的手势识别则成为最重要的技术环节. 本文通过深度卷积神经网络的级联组合, 研究复杂应用场景中第一视角下的一次性学习手势识别(One-shot learning hand gesture recognition, OSLHGR)算法. 考虑到实际应用的便捷性和适用性, 运用改进的轻量级SSD (Single shot multibox detector)目标检测网络实现第一视角下手势目标的快速精确检测; 进而, 以改进的轻量级U-Net网络为主要工具进行复杂背景下手势目标的像素级高效精准分割. 在此基础上, 以组合式3D深度神经网络为工具, 研究提出了一种第一视角下的一次性学习手势动作识别的网络化算法. 在Pascal VOC 2012数据集和SoftKinetic DS325采集的手势数据集上进行的一系列实验测试结果表明, 本文所提出的网络化算法在手势目标检测与分割精度、分类识别准确率和实时性等方面都有显著的优势, 可为在复杂应用环境下实现便捷式高性能智能人机交互提供可靠的技术支持.     

Recognizing human gestures in videos by modeling the mutual context of body position and hands movement

In our days, due the evolution of high-speed computers, the old Human–Computer Interface (HCI) legacies based on mouse and keyboard are slowly becoming obsolete and cannot be accurate enough and respond in a timely manner to the flow of information today. This is why new ways of communicating with the computer have to be researched, the most natural one being the use of gestures. In this paper, a two-level architecture for recognizing human gestures from video frames is proposed. The architecture makes use of several feed-forward neural networks to compute the gestures based on the Haar-like features of body, hand and finger as well as a stochastic-free context grammar that is employed to comprise the mutual context between body pose and hand movement. Trained and tested on 10 gestures (Swipe Right, Swipe Left, Swipe Up, Swipe Down, Horizontal Wave, Vertical Wave, Circle, Point, Palm Up and Fist) the over 94 % accuracy of the system surpasses the current state of the art and compared with a system with no mutual context between body position and hand movement our proposed architecture shows an increase in accuracy with up to 7 %.     

Motion estimation based on optical flow and an artificial neural network (ANN)

Motion estimation provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). Worthy of note is that the visual recognition of hand gestures can help to achieve an easy and natural interaction between human and computer. The interfaces of HCI and other virtual reality systems depend on accurate, real-time hand and fingertip tracking for an association between real objects and the corresponding digital information. However, they are expensive, and complicated operations can make them troublesome. We are developing a real-time, view-based gesture recognition system. The optical flow is estimated and segmented into motion fragments. Using an artificial neural network (ANN), the system can compute and estimate the motions of gestures. Compared with traditional approaches, theoretical and experimental results show that this method has simpler hardware and algorithms, but is more effective. It can be used in moving object recognition systems for understanding human body languages.     

The “Conducting Master”: An Interactive,Real-Time Gesture Monitoring System Based on Spatiotemporal Motion Templates

Research in the field of embodied music cognition has shown the importance of coupled processes of body activity (action) and multimodal representations of these actions (perception) in how music is processed. Technologies in the field of human–computer interaction (HCI) provide excellent means to intervene into, and extend, these coupled action-perception processes. In this article this model is applied to a concrete HCI application, called the “Conducting Master.” The application facilitates multiple users to interact in real time with the system in order to explore and learn how musical meter can be articulated into body movements (i.e., meter-mimicking gestures). Techniques are provided to model and automatically recognize these gestures in order to provide multimodal feedback streams back to the users. These techniques are based on template-based methods that allow approaching meter-mimicking gestures explicitly from a spatiotemporal account. To conclude, some concrete setups are presented in which the functionality of the Conducting Master was evaluated.     

OSG多点触控自然用户接口框架

OSG多点触控自然用户接口框架是在Windows多点触控技术基础上,将触控事件的管理和处理与OSG的事件处理机制相结合,形成了OSG的多点触控运行框架.在与用户交互过程中产生手指触屏的原始数据,根据这些原始数据定义所需的手势,并将其与OSG中交互事件处理机制相结合,完成利用多点触控对三维场景的交互操作.基于以上原理,分析了三维用户交互中主要的操作任务,定义了符合三维空间操作认知的多点触控交互手势,设计了相关手势的识别算法,并通过实例应用的开发验证了这一原理和设计成果的正确性和可行性.     

基于元数据的油气信息管理系统

非结构化数据在信息总量中所占的比例远大于结构化数据,是油气信息管理过程中的宝贵数据资产,如何对这些非结构化数据有效管理和应用是新型油气信息管理系统建设的重要内容之一。为了实现对油气信息资源中包括非结构化数据在内的综合数据高效管理与集成应用,从应用元数据技术角度,一方面对非结构化数据进行定义和详细描述,另一方面以数据库元数据为核心对系统框架进行构建,从而设计了一种新型的油气信息管理系统。系统实现了油气信息综合数据的存储与管理,而且可以对异构数据源进行有效集成应用,有良好的灵活性和扩展性。     

基于多Agent智能网络教学系统设计与实现

针对传统网络教学系统缺乏智能性问题,本文在实现基于XML扩展Agent通信语言基础上,提出并实现了一种基于多Agent智能网络教学系统模型.实际应用说明该系统具有智能性,能提供适应用户的个性化教学服务,激发学生的主观能动性,改善教学效果.     

Intelligent interaction design: the role of human-computer interaction research in the design of intelligent systems

As more intelligent systems are introduced into the marketplace, it is becoming increasingly urgent to consider usability for such systems. Historically, the two fields of artificial intelligence (AI) and human- computer interaction (HCI) have had little in common. In this paper, we consider how established HCI techniques can usefully be applied to the design and evaluation of intelligent systems, and where there is an urgent need for new approaches. Some techniques - notably those for requirements acquisition and empirical evaluation - can usefully be adopted, and indeed are, within many projects. However, many of the tools and techniques developed within HCI to support design and theory-based evaluation cannot be applied in their present forms to intelligent systems because they are based on inappropriate assumptions; there is consequently a need for new approaches. Conversely, there are approaches that have been developed within AI - e.g. in research on dialogue and on ontologies - that could usefully be adapted and encapsulated to respond to this need. These should form the core of a future research agenda for intelligent interaction design.     

Interfacing human and computer with wireless body area sensor networks: the WiMoCA solution

Wireless Body Area Sensor Networks (WBASN) are an emerging technology enabling the design of natural human–computer interfaces (HCI). Automatic recognition of human motion, gestures, and activities is studied in several contexts. For example, mobile computing technology is being considered as a replacement of traditional input systems. Moreover, body posture and activity monitoring can be used for entertainment and health-care applications. However, until now, little work has been done to develop flexible and efficient WBASN solutions suitable for a wide range of applications. Their requirements pose new challenges for sensor network designs, such as optimizing traditional solutions for use as environmental monitoring-like applications and developing on-the-field stress tests. In this paper, we demonstrate the flexibility of a custom-designed WBASN called WiMoCA with respect to a wide range of posture and activity recognition applications by means of practical implementation and on-the-field testing. Nodes of the network mounted on different parts of the human body exploit tri-axial accelerometers to detect its movements. The advanced digital Micro-electro-mechanical system (MEMS) based inertial sensor has been chosen for WiMoCA because it demonstrated high flexibility of use in many different situations, providing the chance to exploit both static and dynamic acceleration components for different purposes. Furthermore, the sensibility and accuracy of the sensing element is perfectly adequate for monitoring human movement, while keeping cost low and size compact, thus meeting our requirements. We implemented three types of applications, stressing the WBASN in many aspects. In fact, they are characterized by different requirements in terms of accuracy, timeliness, and computation distributed on sensing nodes. For each application, we describe its implementation, and we discuss results about performance and power consumption.

Integrating information appliances into an interactive workspace

The authors present a robust, infrastructure-centric, and platform-independent approach to integrating information appliances into the iRoom, an interactive workspace. The Interactive Workspaces Project at Stanford explores new possibilities for people to work together in technology-rich spaces with computing and interaction devices on many different scales. It includes faculty and students from the areas of graphics, human-computer interaction (HCI), networking, ubiquitous computing, and databases, and draws on previous work in all those areas. We design and experiment with multidevice, multiuser environments based on a new architecture that makes it easy to create and add new display and input devices, to move work of all kinds from one computing device to another, and to support and facilitate group interactions. In the same way that today's standard operating systems make it feasible to write single-workstation software that uses multiple devices and networked resources, we are constructing a higher level operating system for the world of ubiquitous computing. We combine research on infrastructure (ways of flexibly configuring and connecting devices, processes, and communication links) with research on HCI (ways of interacting with heterogeneous changing collections of devices with multiple modalities)     

INTELLIGENT SYNTAX-DIRECTED EDITOR THAT MODELS USER PROFICIENCY

Current interactive software systems try to offer many features and a user-friendly interface to aid the user. However, where one system may suit one user, another user may find it difficult to use, because each user is different. User modeling techniques have been applied to make a system suitable for different users. This paper surveys research works for developing interactive systems that use user modeling techniques. After presenting the systems and techniques, we introduce a system called the Intelligent Syntax-Directed Editor (ISE) as an example to show how this approach can benefit the user as an intelligent tool. The ISE assists the user in (1) using the editor more efficiently through ISE's active suggestions and on-line help and (2) program development through the syntax-directed editor. In applying user modeling techniques, ISE builds a profile/model of the user and, based on this model, ISE offers a suitable amount of help and advice depending on the proficiency of the user toward the system. The ISE is implemented on a VAX/780 computer to demonstrate its ability to volunteer advice