Editing and Synthesizing Two‐Character Motions using a Coupled Inverted Pendulum Model

This study aims to develop a controller for use in the online simulation of two interacting characters. This controller is capable of generalizing two sets of interaction motions of the two characters based on the relationships between the characters. The controller can exhibit similar motions to a captured human motion while reacting in a natural way to the opponent character in real time. To achieve this, we propose a new type of physical model called a coupled inverted pendulum on carts that comprises two inverted pendulum on a cart models, one for each individual, which are coupled by a relationship model. The proposed framework is divided into two steps: motion analysis and motion synthesis. Motion analysis is an offline preprocessing step, which optimizes the control parameters to move the proposed model along a motion capture trajectory of two interacting humans. The optimization procedure generates a coupled pendulum trajectory which represents the relationship between two characters for each frame, and is used as a reference in the synthesis step. In the motion synthesis step, a new coupled pendulum trajectory is planned reflecting the effects of the physical interaction, and the captured reference motions are edited based on the planned trajectory produced by the coupled pendulum trajectory generator. To validate the proposed framework, we used a motion capture data set showing two people performing kickboxing. The proposed controller is able to generalize the behaviors of two humans to different situations such as different speeds and turning speeds in a realistic way in real time.     

Performance‐Based Biped Control using a Consumer Depth Camera

We present a technique for controlling physically simulated characters using user inputs from an off‐the‐shelf depth camera. Our controller takes a real‐time stream of user poses as input, and simulates a stream of target poses of a biped based on it. The simulated biped mimics the user's actions while moving forward at a modest speed and maintaining balance. The controller is parameterized over a set of modulated reference motions that aims to cover the range of possible user actions. For real‐time simulation, the best set of control parameters for the current input pose is chosen from the parameterized sets of pre‐computed control parameters via a regression method. By applying the chosen parameters at each moment, the simulated biped can imitate a range of user actions while walking in various interactive scenarios.     

Tunable Robustness: An Artificial Contact Strategy with Virtual Actuator Control for Balance

Physically based characters have not yet received wide adoption in the entertainment industry because control remains both difficult and unreliable. Even with the incorporation of motion capture for reference, which adds believability, characters fail to be convincing in their appearance when the control is not robust. To address these issues, we propose a simple Jacobian transpose torque controller that employs virtual actuators to create a fast and reasonable tracking system for motion capture. We combine this controller with a novel approach we call the topple‐free foot strategy which conservatively applies artificial torques to the standing foot to produce a character that is capable of performing with arbitrary robustness. The system is both easy to implement and straightforward for the animator to adjust to the desired robustness, by considering the trade‐off between physical realism and stability. We showcase the benefit of our system with a wide variety of example simulations, including energetic motions with multiple support contact changes, such as capoeira, as well as an extension that highlights the approach coupled with a Simbicon controlled walker. With this work, we aim to advance the state‐of‐the‐art in the practical design for physically based characters that can employ unaltered reference motion (e.g. motion capture data) and directly adapt it to a simulated environment without the need for optimization or inverse dynamics.     

Dynamics-based analysis and synthesis of human locomotion

One of the best ways to synthesize realistic human motions is to animate characters from captured motion data that inherently respect motion laws. Retargeting and interpolation methods are often used to adapt these motions to different representations of the character and to various environmental constraints but they may introduce physical inaccuracies, although the synthesized motions are natural looking. This paper presents a method for evaluating the physical correctness of retargeted and interpolated locomotions using an inverse dynamics analysis. Furthermore, we propose to improve an initial database with analysed motions that are synthesized again by using a forward dynamics approach. The analysis algorithm consists in determining the resulting forces and torques at joints. With this intention, we develop an automatic creation process of the mass/inertia model of the character. Then using support phase recognition, we compute resulting forces and torques by an inverse dynamics method. The retargeting and the interpolation methods change the physics of the motions. This change is evaluated by using the results of our analysis on artificial and real motions and by using literature results and experimental data from force plates. The evaluation relies on the study of several retargeting and interpolation parameters such as the global size of the character or the structure of the model. The output of this evaluation, the resulting forces and torques at joints, are used to produce physically valid motions by using forward dynamics simulation. With this purpose, we introduce forces and torques normalizations, and finally the synthesized motions may improve the initial database.     

Robust Physics‐based Motion Retargeting with Realistic Body Shapes

Motion capture is often retargeted to new, and sometimes drastically different, characters. When the characters take on realistic human shapes, however, we become more sensitive to the motion looking right. This means adapting it to be consistent with the physical constraints imposed by different body shapes. We show how to take realistic 3D human shapes, approximate them using a simplified representation, and animate them so that they move realistically using physically‐based retargeting. We develop a novel spacetime optimization approach that learns and robustly adapts physical controllers to new bodies and constraints. The approach automatically adapts the motion of the mocap subject to the body shape of a target subject. This motion respects the physical properties of the new body and every body shape results in a different and appropriate movement. This makes it easy to create a varied set of motions from a single mocap sequence by simply varying the characters. In an interactive environment, successful retargeting requires adapting the motion to unexpected external forces. We achieve robustness to such forces using a novel LQR‐tree formulation. We show that the simulated motions look appropriate to each character's anatomy and their actions are robust to perturbations.     

Data‐guided Model Predictive Control Based on Smoothed Contact Dynamics

In this paper, we propose an efficient data‐guided method based on Model Predictive Control (MPC) to synthesize a full‐body motion. Guided by a reference motion, our method repeatedly plans the full‐body motion to produce an optimal control policy for predictive control while sliding the fixed‐span window along the time axis. Based on this policy, the method computes the joint torques of a character at every time step. Together with contact forces and external perturbations if there are any, the joint torques are used to update the state of the character. Without including the contact forces in the control vector, our formulation of the trajectory optimization problem enables automatic adjustment of contact timings and positions for balancing in response to environmental changes and external perturbations. For efficiency, we adopt derivative‐based trajectory optimization on top of state‐of‐the‐art smoothed contact dynamics. Use of derivatives enables our method to run much faster than the existing sampling‐based methods. In order to further accelerate the performance of MPC, we propose efficient numerical differentiation of the system dynamics of a full‐body character based on two schemes: data reuse and data interpolation. The former scheme exploits data dependency to reuse physical quantities of the system dynamics at near‐by time points. The latter scheme allows the use of derivatives at sparse sample points to interpolate those at other time points in the window. We further accelerate evaluation of the system dynamics by exploiting the sparsity of physical quantities such as Jacobian matrix resulting from the tree‐like structure of the articulated body. Through experiments, we show that the proposed method efficiently can synthesize realistic motions such as locomotion, dancing, gymnastic motions, and martial arts at interactive rates using moderate computing resources.     

Simulation Guided Hair Dynamics Modeling from Video

In this paper we present a hybrid approach to reconstruct hair dynamics from multi‐view video sequences, captured under uncontrolled lighting conditions. The key of this method is a refinement approach that combines image‐based reconstruction techniques with physically based hair simulation. Given an initially reconstructed sequence of hair fiber models, we develop a hair dynamics refinement system using particle‐based simulation and incompressible fluid simulation. The system allows us to improve reconstructed hair fiber motions and complete missing fibers caused by occlusion or tracking failure. The refined space‐time hair dynamics are consistent with video inputs and can be also used to generate novel hair animations of different hair styles. We validate this method through various real hair examples.     

Pose Controlled Physically Based Motion

In this paper we describe a new method for generating and controlling physically‐based motion of complex articulated characters. Our goal is to create motion from scratch, where the animator provides a small amount of input and gets in return a highly detailed and physically plausible motion. Our method relieves the animator from the burden of enforcing physical plausibility, but at the same time provides full control over the internal DOFs of the articulated character via a familiar interface. Control over the global DOFs is also provided by supporting kinematic constraints. Unconstrained portions of the motion are generated in real time, since the character is driven by joint torques generated by simple feedback controllers. Although kinematic constraints are satisfied using an iterative search (shooting), this process is typically inexpensive, since it only adjusts a few DOFs at a few time instances. The low expense of the optimization, combined with the ability to generate unconstrained motions in real time yields an efficient and practical tool, which is particularly attractive for high inertia motions with a relatively small number of kinematic constraints.     

Chain Shape Matching for Simulating Complex Hairstyles

Animations of hair dynamics greatly enrich the visual attractiveness of human characters. Traditional simulation techniques handle hair as clumps or continuum for efficiency; however, the visual quality is limited because they cannot represent the fine‐scale motion of individual hair strands. Although a recent mass‐spring approach tackled the problem of simulating the dynamics of every strand of hair, it required a complicated setting of springs and suffered from high computational cost. In this paper, we base the animation of hair on such a fine‐scale on Lattice Shape Matching (LSM), which has been successfully used for simulating deformable objects. Our method regards each strand of hair as a chain of particles, and computes geometrically derived forces for the chain based on shape matching. Each chain of particles is simulated as an individual strand of hair. Our method can easily handle complex hairstyles such as curly or afro styles in a numerically stable way. While our method is not physically based, our GPU‐based simulator achieves visually plausible animations consisting of several tens of thousands of hair strands at interactive rates.     

Real‐time Locomotion Controller using an Inverted‐Pendulum‐based Abstract Model

In this paper, we propose a novel motion controller for the online generation of natural character locomotion that adapts to new situations such as changing user control or applying external forces. This controller continuously estimates the next footstep while walking and running, and automatically switches the stepping strategy based on situational changes. To develop the controller, we devise a new physical model called an inverted‐pendulum‐based abstract model (IPAM). The proposed abstract model represents high‐dimensional character motions, inheriting the naturalness of captured motions by estimating the appropriate footstep location, speed and switching time at every frame. The estimation is achieved by a deep learning based regressor that extracts important features in captured motions. To validate the proposed controller, we train the model using captured motions of a human stopping, walking, and running in a limited space. Then, the motion controller generates human‐like locomotion with continuously varying speeds, transitions between walking and running, and collision response strategies in a cluttered space in real time.     

Data‐Driven Crowd Motion Control With Multi‐Touch Gestures

Controlling a crowd using multi‐touch devices appeals to the computer games and animation industries, as such devices provide a high‐dimensional control signal that can effectively define the crowd formation and movement. However, existing works relying on pre‐defined control schemes require the users to learn a scheme that may not be intuitive. We propose a data‐driven gesture‐based crowd control system, in which the control scheme is learned from example gestures provided by different users. In particular, we build a database with pairwise samples of gestures and crowd motions. To effectively generalize the gesture style of different users, such as the use of different numbers of fingers, we propose a set of gesture features for representing a set of hand gesture trajectories. Similarly, to represent crowd motion trajectories of different numbers of characters over time, we propose a set of crowd motion features that are extracted from a Gaussian mixture model. Given a run‐time gesture, our system extracts the K nearest gestures from the database and interpolates the corresponding crowd motions in order to generate the run‐time control. Our system is accurate and efficient, making it suitable for real‐time applications such as real‐time strategy games and interactive animation controls.     

Motion Sickness Simulation Based on Sensorimotor Control

Sensorimotor control is an essential mechanism for human motions, from involuntary reflex actions to intentional motor skill learning, such as walking, jumping, and swimming. Humans perform various motions according to different task goals and physiological sensory perception; however, most existing computational approaches for motion simulation and generation rarely consider the effects of human perception. The assumption of perfect perception (i.e., no sensory errors) of existing approaches restricts the generated motion types and makes dynamical reactions less realistic. We propose a general framework for sensorimotor control, integrating a balance controller and a vestibular model, to generate perception‐aware motions. By exploiting simulated perception, more natural responses that are closer to human reactions can be generated. For example, motion sickness caused by the impairments in the function of the vestibular system induces postural instability and body sway. Our approach generates physically correct motions and reasonable reactions to external stimuli since the spatial orientation estimation by the vestibular system is essential to preserve balance. We evaluate our framework by demonstrating standing balance on a rotational platform with different angular speeds and duration. The generated motions show that either faster angular speeds or longer rotational duration cause more severe motion sickness. Our results demonstrate that sensorimotor control, integrating human perception and physically‐based control, offers considerable potential for providing more human‐like behaviors, especially for perceptual illusions of human beings, including visual, proprioceptive, and tactile sensations.     

A Layered Particle‐Based Fluid Model for Real‐Time Rendering of Water

We present a physically based real‐time water simulation and rendering method that brings volumetric foam to the real‐time domain, significantly increasing the realism of dynamic fluids. We do this by combining a particle‐based fluid model that is capable of accounting for the formation of foam with a layered rendering approach that is able to account for the volumetric properties of water and foam. Foam formation is simulated through Weber number thresholding. For rendering, we approximate the resulting water and foam volumes by storing their respective boundary surfaces in depth maps. This allows us to calculate the attenuation of light rays that pass through these volumes very efficiently. We also introduce an adaptive curvature flow filter that produces consistent fluid surfaces from particles independent of the viewing distance.     

Topology Aware Data‐Driven Inverse Kinematics

Creating realistic human movement is a time consuming and labour intensive task. The major difficulty is that the user has to edit individual joints while maintaining an overall realistic and collision free posture. Previous research suggests the use of data‐driven inverse kinematics, such that one can focus on the control of a few joints, while the system automatically composes a natural posture. However, as a common problem of kinematics synthesis, penetration of body parts is difficult to avoid in complex movements. In this paper, we propose a new data‐driven inverse kinematics framework that conserves the topology of the synthesizing postures. Our system monitors and regulates the topology changes using the Gauss Linking Integral (GUI), such that penetration can be efficiently prevented. As a result, complex motions with tight body movements, as well as those involving interaction with external objects, can be simulated with minimal manual intervention. Experimental results show that using our system, the user can create high quality human motion in real‐time by controlling a few joints using a mouse or a multi‐touch screen. The movement generated is both realistic and penetration free. Our system is best applied for interactive motion design in computer animations and games.     

Robot control for robust stability with finite reachability time in the whole

A robot model incorporates possible discontinuous nonlinearities with unknown forms and values, unknown payload and unknown predictable external disturbance variations, all in known bounds. A control algorithm is synthesized to guarantee the following: 1.Robust global both stability and attraction with finite reachability time of an appropriately chosen sliding set. 2.The robot motions reach, on the sliding set, a desired motion in a prespecified finite time. 3. Robust both stability and global attraction with finite reachability time of the given robot desired motion. 4. A prespecified convergence quality of real motions to the desired motion, independently of the internal dynamics of the system and without oscillations, hence without chattering in the sliding mode. Robot control robustness means that the controller realizes the control without using information about the real robot internal dynamics. All this is achieved by using the Lyapunov method in a new way combined with a sliding mode approach, but without a variation of the controller structure. The theoretical results are applied to a rotational 3‐degree‐of‐freedom robot. The simulations well verify the robustness of the control algorithm and high quality of robot motions with a prespecified reachability time. ©1999 John Wiley & Sons, Inc.     

State of the Art in Hand and Finger Modeling and Animation

The human hand is a complex biological system able to perform numerous tasks with impressive accuracy and dexterity. Gestures furthermore play an important role in our daily interactions, and humans are particularly skilled at perceiving and interpreting detailed signals in communications. Creating believable hand motions for virtual characters is an important and challenging task. Many new methods have been proposed in the Computer Graphics community within the last years, and significant progress has been made towards creating convincing, detailed hand and finger motions. This state of the art report presents a review of the research in the area of hand and finger modeling and animation. Starting with the biological structure of the hand and its implications for how the hand moves, we discuss current methods in motion capturing hands, data‐driven and physics‐based algorithms to synthesize their motions, and techniques to make the appearance of the hand model surface more realistic. We then focus on areas in which detailed hand motions are crucial such as manipulation and communication. Our report concludes by describing emerging trends and applications for virtual hand animation.     

Physics-driven Multi Dimensional Keyframe Animation for Artist-directable Interactive Character

Various forms of art and entertainment involve many different characters, and advances in human interfaces have necessitated physical interactions in order to develop an improved sense of reality. In this paper we propose a method for generating the motions of characters using multidimensional keyframe animation in parallel with real-time physical simulation. The method generates characters capable of physical interaction, and also allows animators to use traditional methods for designing character motion. We have implemented the system and confirmed its effectiveness experimentally.     

Physically Based Video Editing

Convincing manipulation of objects in live action videos is a difficult and often tedious task. Skilled video editors achieve this with the help of modern professional tools, but complex motions might still lack physical realism since existing tools do not consider the laws of physics. On the other hand, physically based simulation promises a high degree of realism, but typically creates a virtual 3D scene animation rather than returning an edited version of an input live action video. We propose a framework that combines video editing and physics‐based simulation. Our tool assists unskilled users in editing an input image or video while respecting the laws of physics and also leveraging the image content. We first fit a physically based simulation that approximates the object's motion in the input video. We then allow the user to edit the physical parameters of the object, generating a new physical behavior for it. The core of our work is the formulation of an image‐aware constraint within physics simulations. This constraint manifests as external control forces to guide the object in a way that encourages proper texturing at every frame, yet producing physically plausible motions. We demonstrate the generality of our method on a variety of physical interactions: rigid motion, multi‐body collisions, clothes and elastic bodies.     

Motion planning for humanoid robots based on virtual constraints extracted from recorded human movements

In the field of robotics there is a great interest in developing strategies and algorithms to reproduce human-like behavior. In this paper, we consider motion planning for humanoid robots based on the concept of virtual holonomic constraints. At first, recorded kinematic data of particular human motions are analyzed in order to extract consistent geometric relations among various joint angles defining the instantaneous postures. Second, a simplified human body representation leads to dynamics of an underactuated mechanical system with parameters based on anthropometric data. Motion planning for humanoid robots of similar structure can be carried out by considering solutions of reduced dynamics obtained by imposing the virtual holonomic constraints that are found in human movements. The relevance of such a reduced mathematical model in accordance with the real human motions under study is shown. Since the virtual constraints must be imposed on the robot dynamics by feedback control, the design procedure for a suitable controller is briefly discussed.