Copycat hand — robot hand imitating human motions at high speed and with high accuracy

We developed a method enabling them to search similar images at high speed with high accuracy and in uniform processing time, even in the case of using a large-scale database, by clustering databases with uniform numbers of data through self-organizing including self-multiplication and self-extinction, and by collating the input image with the data in the database by means of low-order amounts of characteristics of images, while narrowing the search space in accordance with the past history. By estimating the continuous hand finger shape image string by using the method, we succeeded in realizing processing with an estimation error of the joint angle within several degrees, at a processing time of 150–160 f.p.s, and in an operating time without dispersion, in the case of using a PC having a CPU clock frequency of 2.8 GHz and a memory capacity of 1GB. As the image information and the joint angle information are paired in the database, the robot hand, called the 'Copycat hand', was able to reproduce the same motions as those of the hand and fingers of a human being, without any delay in time, by outputting the results of estimation at the robot hand.     

The interacting effects of forearm rotation and exertion direction on male and female wrist strength

The accurate estimation of wrist strength is an important component of ergonomics task evaluation, as a vast majority of occupational tasks involve use of the hands to generate forces and moments. The purpose of this study was to examine the interacting effects of forearm rotation (pronation/supination) and wrist exertion direction on strength at the wrist joint in males and females. A total of 24 male and female participants performed maximum isometric wrist exertions while maintaining a non-deviated wrist posture (no flexion/extension or radial/ulnar deviation) and an open hand. Maximum wrist moments were obtained in combinations of three forearm rotations (90° pronation, neutral, 90° supination) and four exertion directions (flexion, extension, radial and ulnar deviation). A greater effect of forearm rotation was observed for males, as strength in the neutral forearm posture was significantly different than pronated and supinated postures in 5 of 8 comparisons. For females, both wrist flexion and extension strengths were higher in neutral, compared to supinated forearm postures. The findings of this study suggest that wrist strength does depend on forearm rotation, and this interaction between axes needs to be accounted for in future strength capability estimates.Relevance to industryThis study shows that wrist strength estimates, currently used by ergonomics software packages in industry, can be improved to more accurately reflect the actual wrist strength capabilities of workers during hand-intensive tasks.     

Model-based analysis of hand posture

Researchers have successfully recognized specific fingers of the hand by silhouette images and distinguished a small set of hand signs by contour features of images. However, the silhouette or contour features recovered from the images do not provide sufficient information to generate a 3D hand posture with the fingers positioned properly. This failure led to our study, which developed a new method employing a hand model that can automatically analyze 3D hand postures using static stereo images. Guided by internal constraints and external forces, the model is automatically fitted to the hand image. Our method differs from previous methods in that it captures 27 interacting hand parameters, including finger joint angles, enabling suitable reconstruction of 3D hand posture     

基于深度相机的手腕识别与掌心估测算法(HHME2013-ID51)

当手和手臂都进入深度相机所设定的有效深度范围时,它们将被作为一个整体来提取,若处理时也把它们作为一个整体,这可能会影响手势交互的一些重要算法,如掌心估测,手朝向估测,手的跟踪等。本文分析了手腕的运动特征和手的轮廓特点,并利用内切矩形的几何特征,提出了手腕识别算法来分割手与手臂;为了提高掌心估测算法的性能,本文分析了锐角三角形和最大内切圆的几何特征,并结合手势交互的特点,提出了新的掌心估测算法。经实验证明,手腕识别算法能较好地分割出手与手臂,新的掌心估测算法较之原算法在性能上有显著提高。     

Three-dimensional finger joint angles by hand posture and object properties

The objective of this study was to identify three-dimensional finger joint angles for various hand postures and object properties. Finger joint angles were measured using a VICON system for 10 participants while they pinched objects with two, three, four and five fingers and grasped them with five fingers. The objects were cylinders and square pillars with diameters of 2, 4, 6 and 8 cm and weights of 400, 800, 1400 and 1800 g. Hand posture and object size more significantly affected the joint flexion angles than did object shape and weight. Object shape affected only the metacarpophalangeal (MCP) joint angle of the index finger and the flexion angle of the MCP joint of the little finger. Larger flexion angles resulted when the hand posture was grasping with five fingers. The joint angle increased linearly as the object size decreased. This report provides fundamental information about the specific joint angles of the thumb and fingers.

Practitioner Summary: Three-dimensional finger joint angles are of special interest in ergonomics because of their importance in handheld devices and musculoskeletal hand disorders. In this study, the finger joint angles corresponding to various hand postures and objects with different properties were determined.     

基于傅立叶描述子和HMM的手势识别

针对家庭服务机器人平台中人机交互的问题,提出基于视觉的手势识别作为人与机器人交互的方式,研究利用傅立叶描述子对手势形状进行描述,并结合支持向量机和隐马尔可夫模型分别对静态手势和动态手势进行分类,实现了静态手势和动态手势的识别。该系统基于新型传感器Kinect,在图像分割阶段结合图像深度信息,可以有效的将手势区域提取出来,在一定范围内具有较强的鲁棒性,特征提取阶段基于傅立叶描述子,使手势识别具有旋转、缩放、平移不变性。针对七种常见静态手势和四种动态手势进行测试,平均识别率分别达到98.8%和96.7%,实验结果表明该系统具有较高的准确度。     

基于深度相机的手腕识别与掌心估测

目的当手和手臂都进入深度相机所设定的有效深度范围时,它们将被作为一个整体来提取,若处理时也把它们作为一个整体,这可能会影响手势交互的一些重要算法,如掌心估测、手朝向估测、手的跟踪等。掌心是手势交互中较为稳定的点,掌心与手簇中心的连线常被用来估测手的朝向。因此提高掌心估测算法的性能有助于提高手势交互的整体性能。方法为了有效地分割手与手臂,从分析手腕的运动特征和手的轮廓特点入手,并利用内切矩形的几何特征,提出手腕识别算法;为了提高掌心估测的性能,从手势交互的特点入手,分析了锐角三角形和最大内切圆的几何特征,提出新的掌心估测算法。结果本文算法在空气多点触摸系统中进行了实验,新的掌心估测算法较之原算法在性能上提高了近7倍,且仍然能保持掌心坐标的稳定性,坐标偏差不大于3个像素。同时手腕识别算法的引入也提高了掌心估测的准确性。结论实验结果表明,手腕识别算法能较好地分割出手与手臂,新的掌心估测算法能很好地支持实时交互。     

Wrist postures while keyboarding: effects of a negative slope keyboard system and full motion forearm supports

Abstract

Video-motion analysis was used to analyse hand/wrist posture for subjects typing at a 101-key QWERTY keyboard on a 68 cm high worksurface. Three conditions were tested: subjects typed at the keyboard without arm support, subjects typed with adjustable full motion forearm supports, and subjects typed with an adjustable negative slope keyboard support system. The average declination of the negative slope keyboard support chosen by subjects was 12° below horizontal, which flattened the angle of the key tops. Ulnar deviation was comparable in all conditions and averaged 13° for the right hand and 15° for the left hand. Full motion forearm supports did not significantly affect any postural measures. Dorsal wrist extension averaged 13° when typing with or without the full motion forearm supports, but this was reduced to an average — 1° with the use of the negative slope keyboard support system. Subjects chose to sit at a distance of 79 cm from the computer screen when using the negative slope keyboard system compared with 69 cm without this.     

Visual recognition of continuous hand postures

This paper describes GREFIT (Gesture REcognition based on FInger Tips), a neural network-based system which recognizes continuous hand postures from gray-level video images (posture capturing). Our approach yields a full identification of all finger joint angles (making, however, some assumptions about joint couplings to simplify computations). This allows a full reconstruction of the three-dimensional (3-D) hand shape, using an articulated hand model with 16 segments and 20 joint angles. GREFIT uses a two-stage approach to solve this task. In the first stage, a hierarchical system of artificial neural networks (ANNs) combined with a priori knowledge locates the two-dimensional (2-D) positions of the finger tips in the image. In the second stage, the 2-D position information is transformed by an ANN into an estimate of the 3-D configuration of an articulated hand model, which is also used for visualization. This model is designed according to the dimensions and movement possibilities of a natural human hand. The virtual hand imitates the user's hand to an remarkable accuracy and can follow postures from gray scale images at a frame rate of 10 Hz.     

Interobserver repeatability and validity of an observation method to assess physical loads imposed on the upper extremities

Interobserver repeatability and validity were assessed for a new semiquantitative, time-based observation method for the estimation of physical loads imposed on the upper extremities. Six risk factors of upper extremity disorders were included in the method: repetitive use of hand, use of hand force, pinch grip, non-neutral wrist posture, elevation of upper arm, local mechanical pressure. Two occupational health nurses were trained to use the method. They observed 127 work cycles at a food-processing plant and a paper mill. The method was validated against expert observations from the video, continuous recordings of myoelectric activity (EMG) of forearm muscles, and wrist posture measured with goniometers. Interobserver repeatability was good or moderate for repetitive use of hand, hand force, pinch grip (range κ = 0.58–0.71 on the right; 0.60–0.61 on the left side). Interobserver repeatability was moderate or poor for non-neutral wrist posture, elevation of the upper arm and local mechanical pressure. Validity ranged from moderate to good for repetitive use of hand, use of hand force, pinch grip and non-neutral wrist posture when expert observation was used as reference standard. When observations were validated against force estimations (EMG) and wrist goniometer data, validity was poor. In the absence of generally accepted reference values, arbitrarily chosen limits were used for the proportional duration of some physical load factors. Studies should be carried out to assess the limits that best diVerentiate between safe and hazardous jobs.     

Interobserver repeatability and validity of an observation method to assess physical loads imposed on the upper extremities

Interobserver repeatability and validity were assessed for a new semiquantitative, time-based observation method for the estimation of physical loads imposed on the upper extremities. Six risk factors of upper extremity disorders were included in the method: repetitive use of hand, use of hand force, pinch grip, non-neutral wrist posture, elevation of upper arm, local mechanical pressure. Two occupational health nurses were trained to use the method. They observed 127 work cycles at a food-processing plant and a paper mill. The method was validated against expert observations from the video, continuous recordings of myoelectric activity (EMG) of forearm muscles, and wrist posture measured with goniometers. Interobserver repeatability was good or moderate for repetitive use of hand, hand force, pinch grip (range kappa = 0.58-0.71 on the right; 0.60-0.61 on the left side). Interobserver repeatability was moderate or poor for non-neutral wrist posture, elevation of the upper arm and local mechanical pressure. Validity ranged from moderate to good for repetitive use of hand, use of hand force, pinch grip and non-neutral wrist posture when expert observation was used as reference standard. When observations were validated against force estimations (EMG) and wrist goniometer data, validity was poor. In the absence of generally accepted reference values, arbitrarily chosen limits were used for the proportional duration of some physical load factors. Studies should be carried out to assess the limits that best differentiate between safe and hazardous jobs.     

A new approach for inner-knuckle-print recognition

ObjectiveThis paper proposed a new approach for inner-knuckle-print (IKP) recognition. In traditional IKP recognition systems, the region of interest (ROI) is extracted from the image of the whole hand and the directions of the fingers being imaged are not restricted. The result maybe incorrect because that the shape and surface of the fingers may vary greatly. Moreover, if the direction of the finger being imaged is not restricted, there may be severe rotation transform between intra-class IKPs. To overcome these drawbacks, we develop a new data acquisition scheme as well as an efficient personal authentication algorithm.MethodsThe new scheme is designed to capture the image of the inner surface of the middle knuckles of the middle and ring fingers. The fingers being imaged are kept horizontal with two pegs, so that the rotation angle between different images obtained from the same hand can be minimized. The new personal authentication algorithm consists of the next four steps. Firstly, two regions of interest (ROI), each of which contains the inner surface of a knuckle, are cropped from the original image. Secondly, line features are extracted from the ROIs based on the combination of Gabor filtering and derivative line detection method. Then, binary line images are matched by using a cross-correlation-based method. Finally, the input data is classified through score level fusion.ResultsTo evaluate the proposed IKP recognition system, a finger image database which includes 2000 images from 100 volunteers is established. The images are captured on two separate occasions, at an interval of around two months. Most of the volunteers are not familiar with the image acquisition process. The experimental results show that the proposed system achieves high recognition rate and it works in real time. Moreover, the proposed line feature extraction method outperforms traditional Gabor filter based line detection method and derivative line detection method in accuracy.ConclusionThe proposed IKP system is robust and accurate. It may promote the application and popularization of IKP recognition.     

Wrist posture affects hand and forearm muscle stress during tapping

Non-neutral wrist posture is a risk factor of the musculoskeletal disorders among computer users. This study aimed to assess internal loads on hand and forearm musculature while tapping in different wrist postures. Ten healthy subjects tapped on a key switch using their index finger in four wrist postures: straight, ulnar deviated, flexed and extended. Torque at the finger and wrist joints were calculated from measured joint postures and fingertip force. Muscle stresses of the six finger muscles and four wrist muscles that balanced the calculated joint torques were estimated using a musculoskeletal model and optimization algorithm minimizing the squared sum of muscle stress. Non-neutral wrist postures resulted in greater muscle stresses than the neutral (straight) wrist posture, and the stress in the extensor muscles were greater than the flexors in all conditions. Wrist extensors stress remained higher than 4.5 N/cm² and wrist flexor stress remained below 0.5 N/cm² during tapping. The sustained high motor unit recruitment of extensors suggests a greater risk than other muscles especially in flexed wrist posture. This study demonstrated from the perspective of internal tissue loading the importance of maintaining neutral wrist posture during keying activities.     

Design of optimum grip and control area for one-handed manual control devices

The objective of this paper was to make a design specification of the control area in dual tasks of the hand grip and manual control. The grip postures were analyzed for three types of hand tools. An experiment was performed to measure the position of the fingers and the maximum finger forces at 4 different postures for nine subjects. It was found out that the finger forces were significantly affected by the subjects, the fingers, and the grip postures. The maximum force of women was 62% of men's. From the experiment, the primary control area was defined as 10–13cm and the secondary control area as 8–12cm from the wrist origin. The preferred hand posture of the index and the middle fingers was found to be 3045 degrees at metacarpophalangeal joint and 4050 degrees at proximal interphalangeal joint. It was also found out that the design of one-handed manual control devices should include the characteristics of the user, grip posture, finger force, and the control arrangement.     

The development and validation of equations to predict grip force in the workplace: contributions of muscle activity and posture

The inherent difficulty of measuring forces on the hand in ergonomic workplace assessments has led to the need for equations to predict grip force. A family of equations was developed, and validated, for the prediction of grip force using forearm electromyography (six finger and wrist muscles) as well as posture of the wrist (flexed, neutral and extended) and forearm (pronated, neutral, supinated). Inclusion of muscle activity was necessary to explain over 85% of the grip force variance and was further improved with wrist posture but not forearm posture. Posture itself had little predictive power without muscle activity (<1%). Nominal wrist posture improved predictive power more than the measured wrist angle. Inclusion of baseline muscle activity, the activity required to simply hold the grip dynamometer, greatly improved grip force predictions, especially at low force levels. While the complete model using six muscles and posture was the most accurate, the detailed validation and error analysis revealed that equations based on fewer components often resulted in a negligible reduction in predictive strength. Error was typically less than 10% under 50% of maximal grip force and around 15% over 50% of maximal grip force. This study presents detailed error analyses to both improve upon previous studies and to allow an educated decision to be made on which muscles to monitor depending on expected force levels, costs and error deemed acceptable by the potential user.     

Real-time hand posture recognition using range data

A hand posture recognition system using 3D data is described. The system relies on a novel 3D sensor that generates a dense range image of the scene. The main advantage of the proposed system, compared to other gesture recognition techniques, is the capability for robust unconstrained recognition of complex hand postures such as those encountered in sign language alphabets. This is achieved by explicitly utilizing 3D hand geometry. Moreover, the proposed approach does not rely on color information, and guarantees robust segmentation of the hand under varying illumination conditions, and scene content. Several novel 3D image analysis algorithms are presented, covering the complete processing chain: 3D image acquisition, arm segmentation, hand–forearm segmentation, hand pose estimation, 3D feature extraction, and gesture classification. The proposed system is extensively evaluated.     

融合多尺度特征的复杂手势姿态估计网络

目的 基于单幅RGB图像的手势姿态估计受手势复杂性、手指特征局部自相似性及遮挡问题的影响,导致手势姿态估计准确率低。为此,提出一种面向单目视觉手势姿态估计的多尺度特征融合网络。方法 1）采用ResNet50(50-layer residual network)模块从RGB图像提取不同分辨率特征图,通过通道变换模块显式地学习特征通道间的依赖关系,增强重要的特征通道信息,弱化次要的特征通道信息。2）在全局回归模块中,通过设计节点间的连接方式融合不同分辨率特征图,以便充分利用图像的细节与整体信息。采用局部优化模块继续提取更深层的特征信息,获得手部关节点的高斯热图,以此修正遮挡等原因造成部分关节点回归不准确的问题。3）计算经通道变换模块处理后的最小特征图,通过全局池化和多层感知机处理该特征图以获得手势类别和右手相对于左手的深度。4）综合以上结果获得最终的手势姿态。结果 采用InterHand2.6M和RHD(rendered handpose dataset)数据集训练多尺度特征融合网络,评估指标中根节点的平均误差和关节点的平均误差,均低于同类方法,且在一些复杂和遮挡的场景下鲁棒性更高。在In...     

The effect of viewing angle on wrist posture estimation from photographic images using novice raters

Observational assessment of wrist posture using photographic methods is theoretically affected by camera view angle. A study was conducted to investigate whether wrist flexion/extension and radial/ulnar deviation postures were estimated differently by raters depending on the viewing angle and compared to predictions using a quantitative 2D model of parallax. Novice raters (n = 26) estimated joint angles from images of wrist postures photographed from ten different viewing angles. Results indicated that ideal views, orthogonal to the plane of motion, produced more accurate estimates of posture compared to non-ideal views. The neutral (0°) posture was estimated the most accurately even at different viewing angles. Raters were more accurate than model predictions. Findings demonstrate a need for more systematic methods for collecting and analyzing photographic data for observational studies of posture. Renewed caution in interpreting existing studies of wrist posture where viewing angle was not controlled is advised.     

Effects of combined wrist deviation and forearm rotation on discomfort score

The aim of the study was to examine the pattern of the change in discomfort for combined wrist deviation and forearm rotation as joint angles increased away from neutral in a repetitive task. There were five levels of wrist deviation (neutral, 35% and 55% of the range of motion (ROM) in radial and ulnar deviation) and five levels of forearm rotation (neutral, 30% and 60% of the ROM in pronation and supination). Twenty-five participants performed a repetitive flexion task with a force of 10 N +/- 1 N at a frequency of 15 exertions per min, with replication after 1 week for six of the participants. A visual analogue scale was used for recording the discomfort scores. Repeated measures analysis of covariance with the Greenhouse-Geisser correction, where necessary, was used on transformed values of the discomfort scores. Grip test endurance time at 50% of maximum voluntary contraction was included as a covariate. Wrist deviation (p = 0.007) and forearm rotation (p = 0.001) were found to have significant effects. Interactions of the main factors were not significant and nor was the covariate. Quadratic regression equations were derived and were used to generate iso-discomfort contours, which show a useful area of low discomfort around the central neutral zone of wrist postures, but with steep increases in discomfort at the extreme combinations of wrist ulnar/radial deviation with forearm pronation/supination. Discomfort equations and contours, showing wrist and forearm postures, which are either acceptable or potentially injurious, are useful for the design of industrial tools, machine controls and workspaces. Reference to these can help to reduce the risk of musculoskeletal injury associated with the tasks or tools by avoiding poor postures with unacceptable deviations from neutral posture.     

The development and validation of equations to predict grip force in the workplace: contributions of muscle activity and posture

The inherent difficulty of measuring forces on the hand in ergonomic workplace assessments has led to the need for equations to predict grip force. A family of equations was developed, and validated, for the prediction of grip force using forearm electromyography (six finger and wrist muscles) as well as posture of the wrist (flexed, neutral and extended) and forearm (pronated, neutral, supinated). Inclusion of muscle activity was necessary to explain over 85% of the grip force variance and was further improved with wrist posture but not forearm posture. Posture itself had little predictive power without muscle activity (<1%). Nominal wrist posture improved predictive power more than the measured wrist angle. Inclusion of baseline muscle activity, the activity required to simply hold the grip dynamometer, greatly improved grip force predictions, especially at low force levels. While the complete model using six muscles and posture was the most accurate, the detailed validation and error analysis revealed that equations based on fewer components often resulted in a negligible reduction in predictive strength. Error was typically less than 10% under 50% of maximal grip force and around 15% over 50% of maximal grip force. This study presents detailed error analyses to both improve upon previous studies and to allow an educated decision to be made on which muscles to monitor depending on expected force levels, costs and error deemed acceptable by the potential user.