Self-designed hierarchical network based hand shape intelligent recognition and evaluation for Chinese zither performing

A proper hand shape is the foundation for professional musical instrument performance. In this paper, the image recognition technology based on artificial intelligence is introduced into the hand shape recognition of performing Chinese zither, which is referred to as Zheng in Chinese, for the first time to realize the function of hand shape intelligent evaluation and the self-designed hierarchical network is proposed to recognize and evaluate correct hand shape for Zheng performing. The intra-class difference is larger than the inter-class difference for Zheng performing hand shape image, which belongs to fine-grained image. Therefore, we use the first layer network to determine four classes of images acquired from different viewpoints. Meanwhile, the feature maps from different convolutional blocks of this layer are concatenated as the input of the second layer, which performs fine classification of Zheng performing hand shape images. Consequently, the learning ability of the network can be improved and the complexity of the network can be reduced at the same time. We design an experimental paradigm for instrumentalist hand shape performance evaluation, formulate a Zheng hand shape evaluation merit based on image recognition, and construct a Chinese zither hand shape Dataset (CZ-dataset V3) for the real scene. The experiments show that the method proposed in this paper can effectively improve the recognition accuracy of fine-grained hand shape images and the result is consistent with the evaluation of professional advisors, which realizes the perfect combination of the intelligent image recognition and the hand shape evaluation for Chinese traditional instrument performing.     

Visual recognition of continuous hand postures

This paper describes GREFIT (Gesture REcognition based on FInger Tips), a neural network-based system which recognizes continuous hand postures from gray-level video images (posture capturing). Our approach yields a full identification of all finger joint angles (making, however, some assumptions about joint couplings to simplify computations). This allows a full reconstruction of the three-dimensional (3-D) hand shape, using an articulated hand model with 16 segments and 20 joint angles. GREFIT uses a two-stage approach to solve this task. In the first stage, a hierarchical system of artificial neural networks (ANNs) combined with a priori knowledge locates the two-dimensional (2-D) positions of the finger tips in the image. In the second stage, the 2-D position information is transformed by an ANN into an estimate of the 3-D configuration of an articulated hand model, which is also used for visualization. This model is designed according to the dimensions and movement possibilities of a natural human hand. The virtual hand imitates the user's hand to an remarkable accuracy and can follow postures from gray scale images at a frame rate of 10 Hz.     

Features extraction from hand images based on new detection operators

Human hand shape features extraction from image frame sequences is one of the key steps in human hand 2D/3D tracking system and human hand shape recognition system. In order to satisfy the need of human hand tracking in real time, a fast and accurate method for acquirement of edge features from human hand images without consideration of hand over face is put forward in this paper. The proposed approach is composed of two steps, the coarse location phase (CLP) and the refined location phase (RLP) from coarseness to refinement. In the phase of CLP, the hand contour is approximately described by a polygon with concave and convex, an approach to obtaining hand shape polygon using locating points and locating lines is meticulously discussed. Then, a coarse location (CL) algorithm for extraction of interested hand shape features, such as contour, fingertips, roots of fingers, joints and the intersection of knuckle on different fingers, is proposed. In the phase of RLP, a multi-scale approach is introduced into our study to refine the features obtained by the CL algorithm. By means of defining the response strength of different types of features, a refined location (RL) algorithm is proposed. The major contribution of this paper is that the novel detection operators for features of hand images are presented in the above two steps, which have been successfully applied to our 3D hand shape tracking system and 2D hand shape recognition system. A number of comparative studies with real images and online videos demonstrate that the proposed method can extract the three defined human hand image features with high accuracy and high speed.     

Spatial stimulus response compatibility for a horizontal visual display with hand and foot controls

In this study, two experiments were conducted to examine spatial stimulus–response compatibility (SRC) effects for a horizontal visual display with hand and foot controls. In experiment 1, the hand and foot controls were in a hands above and feet below vertical plane, such that the display and controls were orthogonal to each other. In experiment 2, the foot pedals were moved forward and placed directly underneath the front row of signals, resulting in an additional horizontal display and control relationship. The results of experiment 1 revealed a strong orthogonal SRC effect, such that the best performance was for the front signals with hand controls and rear signals with foot pedals, which was not altered with the presence of horizontal location mappings in experiment 2. These findings indicate that the front-hand/rear-foot mapping relationship was quite robust with regard to changes in the relative locations of the hand and foot response devices.

Statement of Relevance: The results of this study provide useful ergonomics recommendations for designing control consoles with visual signals presented in a horizontal plane and control devices operated by hands and feet. They are helpful for improving efficiency and overall system performance in person–machine systems.     

Optimal hand locations for safe scaffold-end-frame disassembly

Overexertion and fall injuries comprise the largest category of injuries among scaffold workers. A significant portion of these injuries is associated with scaffold-end-frame dismantling tasks, which require both muscle strength and postural balance skills. The commonly used tubular scaffold end frame is 1.52-m wide x 2-m high and weighs 23 kg. Previous studies have indicated that a great muscle strength can be generated when scaffold workers placed their hands symmetrically at knuckle height. However, adequate postural stability can only be reached when the workers placed their hands at the chest or shoulder height, which is near to the height of scaffold-end-frame center-of-mass. A reasonable approach to solve this dilemma is to develop an assistive lifting device, such as a light-weight clip-and-lift bar, that allows workers to place their hands at the height of the center-of-mass of end frames and concurrently allows an optimal hand separation for them to generate an adequate maximum isometric muscle force to safely accomplish the task. This study was conducted to determine the optimal hand location for a conceptual assistive lifting device to mitigate potential postural imbalance while reducing overexertion hazards during scaffold disassembly. This location would be within a window defined by a vertical hand placement between shoulder height and knuckle height and by a horizontal hand separation distance of shoulder width to end-frame width. The whole-body maximum isometric strength of 54 construction workers was measured in nine symmetric scaffold-end-frame disassembly postures, defined by a combination of three vertical hand placements by three horizontal hand separation distances within the aforementioned window. The study apparatus include a computer-controlled data-acquisition system, a custom-fabricated scaffold fixture, and two Bertec force platforms. An analysis of variance showed that the interaction effect of vertical hand placement and hand separation on workers' maximum isometric strength was significant (p < 0.004). A hand location between elbow height and chest height with a hand separation distance of 46 cm (a conceptual, light-weight assistive bar) would allow workers to generate sufficient isometric strength (about twice that of the scaffold weight) to disassemble the typical 23 kg scaffolds while concurrently allowing them to mitigate the likelihood of postural imbalance.     

Non-parametric hand pose estimation with object context

In the spirit of recent work on contextual recognition and estimation, we present a method for estimating the pose of human hands, employing information about the shape of the object in the hand. Despite the fact that most applications of human hand tracking involve grasping and manipulation of objects, the majority of methods in the literature assume a free hand, isolated from the surrounding environment. Occlusion of the hand from grasped objects does in fact often pose a severe challenge to the estimation of hand pose. In the presented method, object occlusion is not only compensated for, it contributes to the pose estimation in a contextual fashion; this without an explicit model of object shape. Our hand tracking method is non-parametric, performing a nearest neighbor search in a large database (.. entries) of hand poses with and without grasped objects. The system that operates in real time, is robust to self occlusions, object occlusions and segmentation errors, and provides full hand pose reconstruction from monocular video. Temporal consistency in hand pose is taken into account, without explicitly tracking the hand in the high-dim pose space. Experiments show the non-parametric method to outperform other state of the art regression methods, while operating at a significantly lower computational cost than comparable model-based hand tracking methods.     

Multi-cue hand detection and tracking for a head-mounted augmented reality system

With the recent developments in wearable augmented reality (AR), the role of natural human–computer interaction is becoming more important. Utilization of auxiliary hardware for interaction introduces extra complexity, weight and cost to wearable AR systems and natural means of interaction such as gestures are therefore more desirable. In this paper, we present a novel multi-cue hand detection and tracking method for head-mounted AR systems which combines depth, color, intensity and curvilinearity. The combination of different cues increases the detection rate, eliminates the background regions and therefore increases the tracking performance under challenging conditions. Detected hand positions and the trajectories are used to perform actions such as click, select, etc. Moreover, the 6 DOF poses of the hands are calculated by approximating the segmented regions with planes in order to render a planar menu (interface) around the hand and use the hand as a planar selection tool. The proposed system is tested on different scenarios (including markers for reference) and the results show that our system can detect and track the hands successfully in challenging conditions such as cluttered and dynamic environments and illumination variance. The proposed hand tracker outperforms other well-known hand trackers under these conditions.     

基于改进PointNet网络的三维手姿估计方法

针对单幅深度图像三维手姿估计中由于手部复杂结构捕捉困难导致的精度低和鲁棒性较差的问题,提出一种基于改进PointNet网络的三维手姿估计方法。该方法首先采用边界框定位网络预测三维边界框,从而准确裁剪手部区域。然后将手部深度图像表示为点云,模拟手部可见表面,有效地利用深度图像中的三维信息。最后将手部点云数据输入改进的PointNet网络,准确地进行三维手姿估计。改进的PointNet网络通过引入跳跃连接,充分利用不同层次的特征,更好地捕捉手部的复杂结构。在NYU手姿数据集上进行验证,实验结果表明,提出的方法优于现有的大部分方法,并且网络结构简单、易于训练,运行速度快。     

Prediction of Manipulation Actions

By looking at a person’s hands, one can often tell what the person is going to do next, how his/her hands are moving and where they will be, because an actor’s intentions shape his/her movement kinematics during action execution. Similarly, active systems with real-time constraints must not simply rely on passive video-segment classification, but they have to continuously update their estimates and predict future actions. In this paper, we study the prediction of dexterous actions. We recorded videos of subjects performing different manipulation actions on the same object, such as “squeezing”, “flipping”, “washing”, “wiping” and “scratching” with a sponge. In psychophysical experiments, we evaluated human observers’ skills in predicting actions from video sequences of different length, depicting the hand movement in the preparation and execution of actions before and after contact with the object. We then developed a recurrent neural network based method for action prediction using as input image patches around the hand. We also used the same formalism to predict the forces on the finger tips using for training synchronized video and force data streams. Evaluations on two new datasets show that our system closely matches human performance in the recognition task, and demonstrate the ability of our algorithms to predict in real time what and how a dexterous action is performed.     

Static hand gesture recognition using neural networks

This paper presents a novel technique for hand gesture recognition through human–computer interaction based on shape analysis. The main objective of this effort is to explore the utility of a neural network-based approach to the recognition of the hand gestures. A unique multi-layer perception of neural network is built for classification by using back-propagation learning algorithm. The goal of static hand gesture recognition is to classify the given hand gesture data represented by some features into some predefined finite number of gesture classes. The proposed system presents a recognition algorithm to recognize a set of six specific static hand gestures, namely: Open, Close, Cut, Paste, Maximize, and Minimize. The hand gesture image is passed through three stages, preprocessing, feature extraction, and classification. In preprocessing stage some operations are applied to extract the hand gesture from its background and prepare the hand gesture image for the feature extraction stage. In the first method, the hand contour is used as a feature which treats scaling and translation of problems (in some cases). The complex moment algorithm is, however, used to describe the hand gesture and treat the rotation problem in addition to the scaling and translation. The algorithm used in a multi-layer neural network classifier which uses back-propagation learning algorithm. The results show that the first method has a performance of 70.83% recognition, while the second method, proposed in this article, has a better performance of 86.38% recognition rate.     

Efficient Deformable Template Detection and Localization without User Initialization

A novel deformable template is presented which detects the boundary of an open hand in a grayscale image without initialization by the user. A dynamic programming algorithm enhanced by pruning techniques finds the hand contour in the image in as little as 19 s on a Pentium 150 MHz. The template is translation- and rotation-invariant and accomodates shape deformation, significant occlusion and background clutter, and the presence of multiple hands.     

Tracking hand rotation and various grasping gestures from an IR camera using extended cylindrical manifold embedding

This paper presents a new approach for tracking hand rotation and various grasping gestures through an infrared camera. For the complexity and ambiguity of an observed hand shape, it is difficult to simultaneously estimate hand configuration and orientation from a silhouette image of a grasping hand gesture. This paper proposes a dynamic shape model for hand grasping gestures using cylindrical manifold embedding to analyze variations of hand shape in different hand configurations between two key hand poses and in simultaneous circular view change by hand rotation. An arbitrary hand shape between two key hand poses from any view can be generated using a cylindrical manifold embedding point after learning nonlinear generative models from the embedding space to the corresponding hand shape observed. The cylindrical manifold embedding model is extended to various grasping gestures by decomposing multiple cylindrical manifold embeddings through grasping style analysis. Grasping hand gestures with simultaneous hand rotation are tracked using particle filters on the manifold space with grasping style estimation. Experimental results for synthetic and real data indicate that the proposed model can accurately track various grasping gestures with hand rotation. The proposed approach may be applied to advanced user interfaces in dark environments by using images beyond the visible spectrum.     

Recognition of dynamic hand gestures

This paper is concerned with the problem of recognition of dynamic hand gestures. We have considered gestures which are sequences of distinct hand poses. In these gestures hand poses can undergo motion and discrete changes. However, continuous deformations of the hand shapes are not permitted. We have developed a recognition engine which can reliably recognize these gestures despite individual variations. The engine also has the ability to detect start and end of gesture sequences in an automated fashion. The recognition strategy uses a combination of static shape recognition (performed using contour discriminant analysis), Kalman filter based hand tracking and a HMM based temporal characterization scheme. The system is fairly robust to background clutter and uses skin color for static shape recognition and tracking. A real time implementation on standard hardware is developed. Experimental results establish the effectiveness of the approach.     

Virtual pottery: a virtual 3D audiovisual interface using natural hand motions

In this paper, we present our approach towards designing and implementing a virtual 3D sound sculpting interface that creates audiovisual results using hand motions in real time. In the interface “Virtual Pottery,” we use the metaphor of pottery creation in order to adopt the natural hand motions to 3D spatial sculpting. Users can create their own pottery pieces by changing the position of their hands in real time, and also generate 3D sound sculptures based on pre-existing rules of music composition. The interface of Virtual Pottery can be categorized by shape design and camera sensing type. This paper describes how we developed the two versions of Virtual Pottery and implemented the technical aspects of the interfaces. Additionally, we investigate the ways of translating hand motions into musical sound. The accuracy of the detection of hand motions is crucial for translating natural hand motions into virtual reality. According to the results of preliminary evaluations, the accuracy of both motion-capture tracking system and portable depth sensing camera is as high as the actual data. We carried out user studies, which took into account information about the two exhibitions along with the various ages of users. Overall, Virtual Pottery serves as a bridge between the virtual environment and traditional art practices, with the consequence that it can lead to the cultivation of the deep potential of virtual musical instruments and future art education programs.     

A versatile interaction framework for robot programming based on hand gestures and poses

This paper proposes a framework for industrial and collaborative robot programming based on the integration of hand gestures and poses. The framework allows operators to control the robot via both End-Effector (EE) and joint movements and to transfer compound shapes accurately to the robot. Seventeen hand gestures, which cover the position and orientation controls of the robotic EE and other auxiliary operations, are designed according to cognitive psychology. Gestures are classified by a deep neural network, which is pre-trained for two-hand pose estimation and fine-tuned on a custom dataset, achieving a test accuracy of 99%. The index finger’s pointing direction and the hand’s orientation are extracted via 3D hand pose estimation to indicate the robotic EE’s moving direction and orientation, respectively. The number of stretched fingers is detected via two-hand pose estimation to represent decimal digits for selecting robot joints and inputting numbers. Finally, we integrate these three manners seamlessly to form a programming framework.We conducted two interaction experiments. The reaction time of the proposed hand gestures in indicating randomly given instructions is significantly less than that of other gesture sets, such as American Sign Language (ASL). The accuracy of our method in compound shape reconstruction is much better than that of hand movement trajectory-based methods, and the operating time is comparable with that of teach pendants.     

An interactive image clipping system using hand motion recognition

We present an efficient hand recognition algorithm for an interactive image clipping system, which is widely used for environments such as public facilities and security environments where personal capturing devices including mobile phones are not allowed. User-friendly interface and accurate image capturing function are required for an image clipping system. We build the system by combining Microsoft Kinect, HD webcam and projector. The Kinect and webcam are used to capture the motions of users׳ hand and project is to display the user-selected area from the capturing material. Hand recognition is composed of three steps: (i) the region occupied by users׳ hand is extracted from an image, (ii) the fingertips of the extracted hand region are analyzed using k-curvature algorithm, and (iii) the height of the fingertip is estimated using the depth image from Kinect. The height of the fingertip informs whether users׳ finger touched the surface of the target. The region captured by the fingertip is clipped from the image and stored as the target image. The excellence of our hand recognition algorithm is proved through a user test.     

Neuroergonomic analysis of perihand space: effects of hand proximity on eye-tracking measures and performance in a visual search task

According to recent studies, cognitive processes are modulated by the proximity of the hands to a stimulus. Specifically, hand proximity (also known as nearby-hand or hand-presence effects) induces a bias to process information near the hands more precisely and this effect can be facilitative or debilitative depending on the task context. Two different distances of the hands in reference to the screen were studied as independent variables: hands placed on the screen and hands placed on the lap. The dependent variables were search times and different eye-tracking parameters. Given the age-related decline in the perception of peripersonal space, the results were analysed for two different age groups. Overall, we found a more detailed evaluation of information near the hands depending on age. In conclusion, the study presents a cognitive behavioural evaluation of human–computer interaction which can be used for touchscreen interface and interaction design as well as modelling human–system interaction.