Real-time Unsupervised Segmentation of human whole-body motion and its application to humanoid robot acquisition of motion symbols

An interactive loop between motion recognition and motion generation is a fundamental mechanism for humans and humanoid robots. We have been developing an intelligent framework for motion recognition and generation based on symbolizing motion primitives. The motion primitives are encoded into Hidden Markov Models (HMMs), which we call “motion symbols”. However, to determine the motion primitives to use as training data for the HMMs, this framework requires a manual segmentation of human motions. Essentially, a humanoid robot is expected to participate in daily life and must learn many motion symbols to adapt to various situations. For this use, manual segmentation is cumbersome and impractical for humanoid robots. In this study, we propose a novel approach to segmentation, the Real-time Unsupervised Segmentation (RUS) method, which comprises three phases. In the first phase, short human movements are encoded into feature HMMs. Seamless human motion can be converted to a sequence of these feature HMMs. In the second phase, the causality between the feature HMMs is extracted. The causality data make it possible to predict movement from observation. In the third phase, movements having a large prediction uncertainty are designated as the boundaries of motion primitives. In this way, human whole-body motion can be segmented into a sequence of motion primitives. This paper also describes an application of RUS to AUtonomous Symbolization of motion primitives (AUS). Each derived motion primitive is classified into an HMM for a motion symbol, and parameters of the HMMs are optimized by using the motion primitives as training data in competitive learning. The HMMs are gradually optimized in such a way that the HMMs can abstract similar motion primitives. We tested the RUS and AUS frameworks on captured human whole-body motions and demonstrated the validity of the proposed framework.     

Construction of a space of motion labels from their mapping to full-body motion symbols

Language is an indispensable for humanoid robot to be integrated into daily life. This paper proposes a novel approach to construct a space of motion labels from their mapping to human whole body motions. The motions are abstracted by Hidden Markov Models, which are referred to as motion symbols. The human motions are automatically partitioned into motion segments, and recognized as sequences of the motion symbols. Sequences of motion labels are also assigned to these motions. The referential relationship between the motion symbols and the motion labels is extracted by stochastic translation model, and distances among the labels are calculated from the association probability of the motion symbols being generated by the labels. The labels are located in a multidimensional space so that the distances are satisfied, and it results in a label space. The label space encapsulates relations among the motion labels such as their similarities. The label space also allows motion recognition. The validity of the constructed label space is demonstrated on a motion capture data-set.     

Hybrid motion graph for character motion synthesis

ObjectiveThis paper proposes a novel framework of Hybrid Motion Graph (HMG) for creating character animations, which enhances the graph-based structural control by motion field representations for efficient motion synthesis of diverse and interactive character animations.MethodsIn HMG framework, the motion template of each class is automatically derived from the training motions for capturing the general spatio-temporal characteristics of an entire motion class. Typical motion field for each class is then constructed. The smooth transitions among motion classes are then generated by interpolating the related motion templates with spacetime constraints. Finally, a hybrid motion graph is built by integrating the separate motion fields for each motion class into the global structural control of motion graph through smooth transition.ResultsIn motion synthesis stage, a character may freely ‘switch’ among different motion classes in the hybrid motion graph via smooth transitions between motion templates and ‘flow’ within each class through the continuous space of motion field with agile and the continuous control process.ConclusionExperimental results show that our framework realizes the fast connectivity among different motion classes and high responsiveness and interactivity for creating realistic character animation of rich behaviors with limited motion data and computational resources.     

Human motion database with a binary tree and node transition graphs

Database of human motion has been widely used for recognizing human motion and synthesizing humanoid motions. In this paper, we propose a data structure for storing and extracting human motion data and demonstrate that the database can be applied to the recognition and motion synthesis problems in robotics. We develop an efficient method for building a human motion database from a collection of continuous, multi-dimensional motion clips. The database consists of a binary tree representing the hierarchical clustering of the states observed in the motion clips, as well as node transition graphs representing the possible transitions among the nodes in the binary tree. Using databases constructed from real human motion data, we demonstrate that the proposed data structure can be used for human motion recognition, state estimation and prediction, and robot motion planning.     

Human motions representation and recognition by directional motion history images

In this paper, we describe a technique for representing and recognizing human motions using directional motion history images. A motion history image is a single human motion image produced by superposing binarized successive motion image frames so that older frames may have smaller weights. It has, however, difficulty that the latest motion overwrites older motions, resulting in inexact motion representation and therefore incorrect recognition. To overcome this difficulty, we propose directional motion history images which describe a motion with respect to four directions of movement, i.e. up, down, right and left, employing optical flow. The directional motion history images are thus a set of four motion history images defined on four optical flow images. Experimental results show that the proposed technique achieves better performance in the recognition of human motions than the existent motion history images. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

Enriching a motion database by analogous combination of partial human motions

We have synthesized new human body motions from existing motion data, by dividing the body of an animated character into several parts, such as upper and lower body, and partitioning the motion of the character into corresponding partial motions. By combining different partial motions, we can generate new motion sequences. We select the most natural-looking combinations by analyzing the similarity of partial motions, using techniques such as motion segmentation, dimensionality reduction, and clustering. These new combinations can dramatically increase the size of a motion database, allowing more score in selecting motions to meet constraints, such as collision avoidance. We verify the naturalness and physical plausibility of the new motions using an SVM learning model and by analysis of static and dynamic balance.     

Human Motion Synthesis with Optimization‐based Graphs

Continuous constrained optimization is a powerful tool for synthesizing novel human motion segments that are short. Graph‐based motion synthesis methods such as motion graphs and move trees are popular ways to synthesize long motions by playing back a sequence of existing motion segments. However, motion graphs only support transitions between similar frames, and move trees only support transitions between the end of one motion segment and the start of another. In this paper, we introduce an optimization‐based graph that combines continuous constrained optimization with graph‐based motion synthesis. The constrained optimization is used to create a vast number of complex realistic‐looking transitions in the graph. The graph can then be used to synthesize long motions with non‐trivial transitions that for example allow the character to switch its behavior abruptly while retaining motion naturalness. We also propose to build this graph semi‐autonomously by requiring a user to classify generated transitions as acceptable or not and explicitly minimizing the amount of required classifications. This process guarantees the quality consistency of the optimization‐based graph at the cost of limited user involvement.     

A real-time system for motion retrieval and interpretation

This paper proposes a new examplar-based method for real-time human motion recognition using Motion Capture (MoCap) data. We have formalized streamed recognizable actions, coming from an online MoCap engine, into a motion graph that is similar to an animation motion graph. This graph is used as an automaton to recognize known actions as well as to add new ones. We have defined and used a spatio-temporal metric for similarity measurements to achieve more accurate feedbacks on classification. The proposed method has the advantage of being linear and incremental, making the recognition process very fast and the addition of a new action straightforward. Furthermore, actions can be recognized with a score even before they are fully completed. Thanks to the use of a skeleton-centric coordinate system, our recognition method has become view-invariant. We have successfully tested our action recognition method on both synthetic and real data. We have also compared our results with four state-of-the-art methods using three well known datasets for human action recognition. In particular, the comparisons have clearly shown the advantage of our method through better recognition rates.     

Toward memory-based human motion simulation: development and validation of a motion modification algorithm

Computer simulation of human motions helps test hypotheses on human motion planning and fosters timely and high-quality human-machine/environment interaction design. The current study introduces a novel simulation approach termed memory-based motion simulation (MBMS), and presents its key element "motion modification" (MoM) algorithm. The proposed approach implements a computational model inspired by the generalized motor program (GMP) theory. Operationally, when a novel motion scenario is submitted to the MBMS system, its motion database is searched to find relevant existing motions. The selected motions, referred to as "root motions", most likely do not meet exactly the novel motion scenario, and therefore, they need to be modified by the MoM algorithm. This algorithm derives a parametric representation of possible variants of a root motion in a GMP-like manner, and adjusts the parameter values such that the new modified motion satisfies the novel motion scenario, while retaining the root motion's overall angular movement pattern and inter-joint coordination. An evaluation of the prediction capability of the algorithm, using both seated upper body reaching and whole-body load-transfer motions, indicated that the algorithm can accurately predict various human motions with errors comparable to the inherent variability in human motions when repeated under identical task conditions.     

Creating and retargetting motion by the musculoskeletal human body model

Recently, optimization has been used in various ways to interpolate or retarget human body motions obtained by motion-capturing systems. However, in such cases, the inner structure of a human body has rarely been taken into account, and hence there have been difficulties in simulating physiological effects such as fatigue or injuries. In this paper, we propose a method to create/retarget human body motions using a musculoskeletal human body model. Using our method, it is possible to create dynamically and physiologically feasible motions. Since a muscle model based on Hill's model is included in our system, it is also possible to retarget the original motion by changing muscular parameters. For example, using the muscle fatigue model, a motion where a human body gradually gets tired can be simulated. By increasing the maximal force exertable by the muscles, or decreasing it to zero, training or displacement effects of muscles can also be simulated. Our method can be used for biomechanically correct inverse kinematics, interpolation of motions, and physiological retargetting of the human body motion.     

Statistics-based Motion Synthesis for Social Conversations

Plausible conversations among characters are required to generate the ambiance of social settings such as a restaurant, hotel lobby, or cocktail party. In this paper, we propose a motion synthesis technique that can rapidly generate animated motion for characters engaged in two-party conversations. Our system synthesizes gestures and other body motions for dyadic conversations that synchronize with novel input audio clips. Human conversations feature many different forms of coordination and synchronization. For example, speakers use hand gestures to emphasize important points, and listeners often nod in agreement or acknowledgment. To achieve the desired degree of realism, our method first constructs a motion graph that preserves the statistics of a database of recorded conversations performed by a pair of actors. This graph is then used to search for a motion sequence that respects three forms of audio-motion coordination in human conversations: coordination to phonemic clause, listener response, and partner's hesitation pause. We assess the quality of the generated animations through a user study that compares them to the originally recorded motion and evaluate the effects of each type of audio-motion coordination via ablation studies.     

Path planning directed motion control of virtual humans in complex environments

Natural motion synthesis of virtual humans have been studied extensively, however, motion control of virtual characters actively responding to complex dynamic environments is still a challenging task in computer animation. It is a labor and cost intensive animator-driven work to create realistic human motions of character animations in a dynamically varying environment in movies, television and video games. To solve this problem, in this paper we propose a novel approach of motion synthesis that applies the optimal path planning to direct motion synthesis for generating realistic character motions in response to complex dynamic environment. In our framework, SIPP (Safe Interval Path Planning) search is implemented to plan a globally optimal path in complex dynamic environments. Three types of control anchors to motion synthesis are for the first time defined and extracted on the obtained planning path, including turning anchors, height anchors and time anchors. Directed by these control anchors, highly interactive motions of virtual character are synthesized by motion field which produces a wide variety of natural motions and has high control agility to handle complex dynamic environments. Experimental results have proven that our framework is capable of synthesizing motions of virtual humans naturally adapted to the complex dynamic environments which guarantee both the optimal path and the realistic motion simultaneously.     

A semantic tree algorithm for the generation of sextet polynomials of hexagonal systems

The sextet polynomial that counts different ways of selecting varying number of resonating sextets on the hexagonal system is computed using a search based symbol manipulation algorithm. This is a #P Complete combinatorial enumeration problem, and artificial intelligence (AI) is employed for efficient enumeration. This is done by selective exploration of the semantic tree defined for that purpose. Hexagons of the graph are defined as symbols and each node of the tree is defined as a set of mutually disjoint hexagon patterns of the graph. The sextet polynomial is generated by enumerating a suitable subset of the nodes of the tree. A pruning heuristic that avoids redundant branches by a priori learning at selected intelligent branches of the semantic tree is designed.     

Semantic Plane-Structure based motion detection with a nonstationary camera

This paper presents a novel method to accurately detect moving objects from a video sequence captured using a nonstationary camera. Although common methods provide effective motion detection for static backgrounds or through only planar-perspective transformation, many detection errors occur when the background contains complex dynamic interferences or the camera undergoes unknown motions. To solve this problem, this study proposed a motion detection method that incorporates temporal motion and spatial structure. In the proposed method, first, spatial semantic planes are segmented, and image registration based on stable background planes is applied to overcome the interferences of the foreground and dynamic background. Thus, the estimated dense temporal motion ensures that small moving objects are not missed. Second, motion pixels are mapped on semantic planes, and then, the spatial distribution constraints of motion pixels, regional shapes and plane semantics, which are integrated into a planar structure, are used to minimise false positives. Finally, based on the dense temporal motion and spatial structure, moving objects are accurately detected. The experimental results on CDnet dataset, Pbi dataset, Aeroscapes dataset, and other challenging self-captured videos under difficult conditions, such as fast camera movement, large zoom variation, video jitters, and dynamic background, revealed that the proposed method can remove background movements, dynamic interferences, and marginal noises and can effectively obtain complete moving objects.© 2017 ElsevierInc.Allrightsreserved.     

Achieving good connectivity in motion graphs

Motion graphs have been widely successful in the synthesis of human motions. However, the quality of the generated motions depends heavily on the connectivity of the graphs and the quality of transitions in them. Achieving both of these criteria simultaneously though is difficult. Good connectivity requires transitions between less similar poses, while good motion quality requires transitions only between very similar poses. This paper introduces a new method for building motion graphs. The method first builds a set of interpolated motion clips, which contains many more similar poses than the original data set. The method then constructs a well-connected motion graph (wcMG), by using as little of the interpolated motion clip frames as necessary to provide good connectivity and only smooth transitions. Based on experiments, wcMGs outperform standard motion graphs across different measures, generate good quality motions, allow for high responsiveness in interactive control applications, and do not even require post-processing of the synthesized motions.     

The human motion database: A cognitive and parametric sampling of human motion

Motion databases have a strong potential to guide progress in the field of machine recognition and motion-based animation. Existing databases either have a very loose structure that does not sample the domain according to any controlled methodology or too few action samples which limit their potential to quantitatively evaluate the performance of motion-based techniques. The controlled sampling of the motor domain in the database may lead investigators to identify the fundamental difficulties of motion cognition problems and allow the addressing of these issues in a more objective way. In this paper, we describe the construction of our Human Motion Database using controlled sampling methods (parametric and cognitive sampling) to obtain the structure necessary for the quantitative evaluation of several motion-based research problems. The Human Motion Database is organized into several components: the praxicon dataset, the cross-validation dataset, the generalization dataset, the compositionality dataset, and the interaction dataset. The main contributions of this paper include (1) a survey of human motion databases describing data sources related to motion synthesis and analysis problems, (2) a sampling methodology that takes advantage of a systematic controlled capture, denoted as cognitive sampling and parametric sampling, and (3) a novel structured motion database organized into several datasets addressing a number of aspects in the motion domain.     

融合时空图卷积网络与非自回归模型的三维人体运动预测

当前人体运动预测的方法大多采用基于图卷积网络的自回归模型,没有充分考虑关节间的特有关系和自回归网络性能的限制,从而产生平均姿态和误差累积等问题。为解决以上问题,提出融合时空图卷积网络和非自回归的模型对人体运动进行预测。一方面利用时空图卷积的网络提取人体运动序列的局部特征,可以有效减少三维人体运动预测场景中的平均姿态问题和过度堆叠图卷积层引起的过平滑问题的发生;另一方面将非自回归模型与时空图卷积网络进行结合,减少误差累计问题的发生。利用Human3.6M的数据集进行80 ms、160 ms、320 ms和400 ms的人体运动预测实验。结果表明,NAS-GCN模型与现有方法相比,能预测出更精确的结果。     

An interactive data-driven driving simulator using motion blending

Compared to the motion equations the data-driven method can simulate reality from sampling of real motions but real-time interaction between a user and the simulator is problematic. Existing data-driven motion generation methods simply record and replay the motion of the vehicle. Character animation technology enables a user to control motions that are generated by a motion capture database and an appropriate motion control algorithm. We propose a data-driven motion generation method and implement a driving simulator by adapting the method of motion capture. The motion data sampled from a real vehicle are transformed into appropriate data structures called motion blocks, and then a series of motion blocks are saved into the motion database. During simulation, the driving simulator searches for and synthesizes optimal motion blocks from the motion database and generates motion streams that reflect the current simulation conditions and parameterized user demands. We demonstrate the proposed method through experiments with the driving simulator.