A Survey on Human Performance Capture and Animation

With the rapid development of computing technology, three-dimensional (3D) human body models and their dynamic motions are widely used in the digital entertainment industry. Human performance mainly involves human body shapes and motions. Key research problems in human performance animation include how to capture and analyze static geometric appearance and dynamic movement of human bodies, and how to simulate human body motions with physical effects. In this survey, according to the main research directions of human body performance capture and animation, we summarize recent advances in key research topics, namely human body surface reconstruction, motion capture and synthesis, as well as physics-based motion simulation, and further discuss future research problems and directions. We hope this will be helpful for readers to have a comprehensive understanding of human performance capture and animation.     

Design and Use of Linear Models for Image Motion Analysis

Linear parameterized models of optical flow, particularly affine models, have become widespread in image motion analysis. The linear model coefficients are straightforward to estimate, and they provide reliable estimates of the optical flow of smooth surfaces. Here we explore the use of parameterized motion models that represent much more varied and complex motions. Our goals are threefold: to construct linear bases for complex motion phenomena; to estimate the coefficients of these linear models; and to recognize or classify image motions from the estimated coefficients. We consider two broad classes of motions: i) generic motion features such as motion discontinuities and moving bars; and ii) non-rigid, object-specific, motions such as the motion of human mouths. For motion features we construct a basis of steerable flow fields that approximate the motion features. For object-specific motions we construct basis flow fields from example motions using principal component analysis. In both cases, the model coefficients can be estimated directly from spatiotemporal image derivatives with a robust, multi-resolution scheme. Finally, we show how these model coefficients can be use to detect and recognize specific motions such as occlusion boundaries and facial expressions.     

Simultaneous particle tracking in multi-action motion models with synthesized paths

This paper proposes human motion models of multiple actions for 3D pose tracking. A training pose sequence of each action, such as walking and jogging, is separately recorded by a motion capture system and modeled independently. This independent modeling of action-specific motions allows us 1) to optimize each model in accordance with only its respective motion and 2) to improve the scalability of the models. Unlike existing approaches with similar motion models (e.g. switching dynamical models), our pose tracking method uses the multiple models simultaneously for coping with ambiguous motions. For robust tracking with the multiple models, particle filtering is employed so that particles are distributed simultaneously in the models. Efficient use of the particles can be achieved by locating many particles in the model corresponding to an action that is currently observed. For transferring the particles among the models in quick response to changes in the action, transition paths are synthesized between the different models in order to virtually prepare inter-action motions. Experimental results demonstrate that the proposed models improve accuracy in pose tracking.     

Multibody grouping by inference of multiple subspaces from high-dimensional data using oriented-frames

Recently, subspace constraints have been widely exploited in many computer vision problems such as multibody grouping. Under linear projection models, feature points associated with multiple bodies reside in multiple subspaces. Most existing factorization-based algorithms can segment objects undergoing independent motions. However, intersections among the correlated motion subspaces will lead most previous factorization-based algorithms to erroneous segmentation. To overcome this limitation, in this paper, we formulate the problem of multibody grouping as inference of multiple subspaces from a high-dimensional data space. A novel and robust algorithm is proposed to capture the configuration of the multiple subspace structure and to find the segmentation of objects by clustering the feature points into these inferred subspaces, no matter whether they are independent or correlated. In the proposed method, an oriented-frame (OF), which is a multidimensional coordinate frame, is associated with each data point indicating the point's preferred subspace configuration. Based on the similarity between the subspaces, novel mechanisms of subspace evolution and voting are developed. By filtering the outliers due to their structural incompatibility, the subspace configurations will emerge. Compared with most existing factorization-based algorithms that cannot correctly segment correlated motions, such as motions of articulated objects, the proposed method has a robust performance in both independent and correlated motion segmentation. A number of controlled and real experiments show the effectiveness of the proposed method. However, the current approach does not deal with transparent motions and motion subspaces of different dimensions.     

A hybrid dynamic motion prediction method for multibody digital human models based on a motion database and motion knowledge

In this paper, we present a novel method to predict human motion, seeking to combine the advantages of both data-based and knowledge-based motion prediction methods. Our method relies on a database of captured motions for reference and introduces knowledge in the prediction in the form of a motion control law, which is followed while resembling the actually performed reference motion. The prediction is carried out by solving an optimization problem in which the following conditions are imposed to the motion: must fulfill the goals of the task; resemble the reference motion selected from the database; follow a knowledge-based dynamic motion control law; and ensure the dynamic equilibrium of the human model, considering its interactions with the environment. In this work, we apply the proposed method to a database of clutch pedal depression motions, and we present the results for three predictions. The method is validated by comparing the results of the prediction to motions actually performed in similar conditions. The predicted motions closely resemble the motions in the validation database and no significant differences have been noted either in the motion’s kinematics or in the motion’s dynamics.     

Generating 3D architectural models based on hand motion and gesture

This paper presents a novel method for rapidly generating 3D architectural models based on hand motion and design gestures captured by a motion capture system. A set of sign language-based gestures, architectural hand signs (AHS), has been developed. AHS is performed on the left hand to define various “components of architecture”, while “location, size and shape” information is defined by the motion of Marker-Pen on the right hand. The hand gestures and motions are recognized by the system and then transferred into 3D curves and surfaces correspondingly. This paper demonstrates the hand gesture-aided architectural modeling method with some case studies.     

Hybrid predictive dynamics: a new approach to simulate human motion

A new methodology, called hybrid predictive dynamics (HPD), is introduced in this work to simulate human motion. HPD is defined as an optimization-based motion prediction approach in which the joint angle control points are unknowns in the equations of motion. Some of these control points are bounded by the experimental data. The joint torque and ground reaction forces are calculated by an inverse algorithm in the optimization procedure. Therefore, the proposed method is able to incorporate motion capture data into the formulation to predict natural and subject-specific human motions. Hybrid predictive dynamics includes three procedures, and each is a sub-optimization problem. First, the motion capture data are transferred from Cartesian space into joint space by using optimization-based inverse kinematics (IK) methodology. Secondly, joint profiles obtained from IK are interpolated by B-spline control points by using an error-minimization algorithm. Third, boundaries are built on the control points to represent specific joint profiles from experiments, and these boundaries are used to guide the predicted human motion. To predict more accurate motion, the boundaries can also be built on the kinetic variables if the experimental data are available. The efficiency of the method is demonstrated by simulating a box-lifting motion. The proposed method takes advantage of both prediction and tracking capabilities simultaneously, so that HPD has more applications in human motion prediction, especially towards clinical applications.     

Memory-Based Human Motion Simulation for Computer-Aided Ergonomic Design

A novel memory-based motion simulation (MBMS) model was developed as a general framework for simulating natural human motions for computer-aided ergonomic design. The MBMS model utilizes real human motion samples recorded in motion capture experiments as templates for simulating novel motions. Such human motion samples are stored in a motion database. When a user submits an input simulation scenario to the model, a motion search engine termed the ldquoroot motion finderrdquo in the model searches the motion database and retrieves the motion samples that closely match the given scenario. The retrieved motions, referred to as root motions, may significantly differ from one another in the underlying movement technique. Such variability within the root motion set is analyzed and graphically summarized by a model component termed the motion variability analyzer. This analysis helps users rapidly identify alternative movement techniques for the given input simulation scenario and simulate human motions based on alternative movement techniques. Since root motions do not exactly satisfy but only closely match the input simulation scenario, a motion modification (MoM) algorithm adapts them to fit the scenario by systematically deforming them in the joint angle-time domain. The MoM algorithm retains the root motions' fundamental spatial-temporal structure and minimizes deviations from the root motions during such deformations. The MBMS model overcomes limitations of existing simulation models and achieves the following: 1) simulation of categorically different motions based on a single unified model; 2) simple and efficient learning of new motion behaviors; and 3) representation and simulation of human motion variability.     

采用弱预测机制的人体运动跟踪算法

快速运动和自遮挡是人体运动跟踪的难点所在 .为此提出了一种采用弱预测机制的人体运动跟踪算法 .该算法首先通过全局搜索 ,确定候选人体特征集 ;然后建立特征的色彩、运动等属性的时变模型 ,构造贝叶斯分类器 ,实现特征对应 ;最后根据人体特征层次模型 ,检验特征匹配 ,并实现被遮挡特征的定位 .为提高跟踪效率 ,采用了基于图象多分辨率表示的特征搜索算法 ,由低分辨率图象通过全局搜索来获取初始候选特征集 ,然后在高分辨率下 ,不断改善候选特征精度 .实验结果表明 ,该算法能实现对快速人体运动的跟踪并有效解决自遮挡问题 .     

Searching for Complex Human Activities with No Visual Examples

We describe a method of representing human activities that allows a collection of motions to be queried without examples, using a simple and effective query language. Our approach is based on units of activity at segments of the body, that can be composed across space and across the body to produce complex queries. The presence of search units is inferred automatically by tracking the body, lifting the tracks to 3D and comparing to models trained using motion capture data. Our models of short time scale limb behaviour are built using labelled motion capture set. We show results for a large range of queries applied to a collection of complex motion and activity. We compare with discriminative methods applied to tracker data; our method offers significantly improved performance. We show experimental evidence that our method is robust to view direction and is unaffected by some important changes of clothing.     

Extraction of bodily features for gait recognition and gait attractiveness evaluation

Although there has been much previous research on which bodily features are most important in gait analysis, the questions of which features should be extracted from gait, and why these features in particular should be extracted, have not been convincingly answered. The primary goal of the study reported here was to take an analytical approach to answering these questions, in the context of identifying the features that are most important for gait recognition and gait attractiveness evaluation. Using precise 3D gait motion data obtained from motion capture, we analyzed the relative motions from different body segments to a root marker (located on the lower back) of 30 males by the fixed root method, and compared them with the original motions without fixing root. Some particular features were obtained by principal component analysis (PCA). The left lower arm, lower legs and hips were identified as important features for gait recognition. For gait attractiveness evaluation, the lower legs were recognized as important features.     

Self‐similarity Analysis for Motion Capture Cleaning

Motion capture sequences may contain erroneous data, especially when the motion is complex or performers are interacting closely and occlusions are frequent. Common practice is to have specialists visually detect the abnormalities and fix them manually. In this paper, we present a method to automatically analyze and fix motion capture sequences by using self‐similarity analysis. The premise of this work is that human motion data has a high‐degree of self‐similarity. Therefore, given enough motion data, erroneous motions are distinct when compared to other motions. We utilize motion‐words that consist of short sequences of transformations of groups of joints around a given motion frame. We search for the K‐nearest neighbors (KNN) set of each word using dynamic time warping and use it to detect and fix erroneous motions automatically. We demonstrate the effectiveness of our method in various examples, and evaluate by comparing to alternative methods and to manual cleaning.     

基于学习的群体动画生成技术研究

为了降低群体动画中生成大量自然而又相似的人体运动的难度和复杂性,研究了一种基于学习的群体动画生成技术。该技术首先通过建立基于高斯过程隐变量模型和隐空间动态模型的运动姿势学习模型,将高维运动姿势映射到低维隐空间中,并在低维隐空间对相邻姿势的动态演化进行建模;然后通过对已有运动数据的学习来获得组成该运动的姿势的概率分布,再通过隐空间中的动态预测和Hybrid Monte Carlo采样来得到符合给定概率分布的隐轨迹;最后通过姿势重构来得到与原运动非常相似但又不同的一系列自然的运动,以产生群体动画,从而避开了传统的基于几何和物理约束的逆运动方法固有的困难和复杂性。     

Maximum acceptable forces for repetitive wrist extension with a pinch grip

This study represents a continuation of a series of psychophysical studies on repetitive motions of the wrist conducted at the Liberty Mutual Research Center for Safety and Health. The purpose of the study was to quantify maximum acceptable forces for extension motions of the wrist performed with a pinch grip. Subjects grasped a handle with a pinch grip and moved it through a 1.57 rad (90°) extension wrist motion (similar to a light assembly operation). A psychophysical methodology was used in which the subject adjusted the resistance on the handle, and the experimenter manipulated or controlled all other variables. Twenty subjects performed the task at repetition rates of 15, 20 and 25 motions per minute. Subjects performed for 7 h per day, 5 days per week, for 4 weeks. The subjects were instructed to work as if they were on an incentive basis, getting paid for the amount of work they performed. Symptoms were recorded by the subjects during the last 5 min of each hour. The results are presented and compared with maximum acceptable forces for other types of wrist motion investigated in previous studies. Maximum acceptable force for wrist extension with a pinch grip is smaller than any of the other motions investigated so far.

Relevance to industry

Cumulative trauma disorders of the upper extremities continue to be a problem for industrial workers who perform repetitive tasks. Although a number of physical risk factors have been identified, there are very few data available for establishing acceptable levels of these risk factors. This study attempted to collect such data.     

Improving digital human modelling for proactive ergonomics in design

This paper presents the need to improve existing digital human models (DHMs) so they are better able to serve as effective ergonomics analysis and design tools. Existing DHMs are meant to be used by a designer early in a product development process when attempting to improve the physical design of vehicle interiors and manufacturing workplaces. The emphasis in this paper is placed on developing future DHMs that include valid posture and motion prediction models for various populations. It is argued that existing posture and motion prediction models now used in DHMs must be changed to become based on real motion data to assure validity for complex dynamic task simulations. It is further speculated that if valid human posture and motion prediction models are developed and used, these can be combined with psychophysical and biomechanical models to provide a much greater understanding of dynamic human performance and population specific limitations and that these new DHM models will ultimately provide a powerful ergonomics design tool.     

Improving digital human modelling for proactive ergonomics in design

This paper presents the need to improve existing digital human models (DHMs) so they are better able to serve as effective ergonomics analysis and design tools. Existing DHMs are meant to be used by a designer early in a product development process when attempting to improve the physical design of vehicle interiors and manufacturing workplaces. The emphasis in this paper is placed on developing future DHMs that include valid posture and motion prediction models for various populations. It is argued that existing posture and motion prediction models now used in DHMs must be changed to become based on real motion data to assure validity for complex dynamic task simulations. It is further speculated that if valid human posture and motion prediction models are developed and used, these can be combined with psychophysical and biomechanical models to provide a much greater understanding of dynamic human performance and population specific limitations and that these new DHM models will ultimately provide a powerful ergonomics design tool.     

Construction of a space of motion labels from their mapping to full-body motion symbols

Language is an indispensable for humanoid robot to be integrated into daily life. This paper proposes a novel approach to construct a space of motion labels from their mapping to human whole body motions. The motions are abstracted by Hidden Markov Models, which are referred to as motion symbols. The human motions are automatically partitioned into motion segments, and recognized as sequences of the motion symbols. Sequences of motion labels are also assigned to these motions. The referential relationship between the motion symbols and the motion labels is extracted by stochastic translation model, and distances among the labels are calculated from the association probability of the motion symbols being generated by the labels. The labels are located in a multidimensional space so that the distances are satisfied, and it results in a label space. The label space encapsulates relations among the motion labels such as their similarities. The label space also allows motion recognition. The validity of the constructed label space is demonstrated on a motion capture data-set.     

Wii remote–based low-cost motion capture for automated assembly simulation

This paper describes the development of a Wii remote (Wiimote)–based low-cost motion capture system and demonstrates its application for automated assembly simulation. Multiple Wiimotes are used to form a vision system to perform motion capture in 3D space. A hybrid algorithm for calibrating a multi-camera stereo vision system has been developed based on Zhang’s and Svoboda’s calibration algorithms. This hybrid algorithm has been evaluated and shown accuracy improvement over Svoboda’s algorithm for motion capture with multiple cameras. The captured motion data are used to automatically generate an assembly simulation of objects represented by CAD models in real time. The Wiimote-based motion capture system is practically attractive because it is inexpensive, wireless, and easily portable. Application examples have been developed for a single vision system with two Wiimotes to track the assembly of a microsatellite prototype frame and for an integrated vision system with four Wiimotes to track the assembly of a bookshelf.