Aura Mesh: Motion Retargeting to Preserve the Spatial Relationships between Skinned Characters

Applying motion‐capture data to multi‐person interaction between virtual characters is challenging because one needs to preserve the interaction semantics while also satisfying the general requirements of motion retargeting, such as preventing penetration and preserving naturalness. An efficient means of representing interaction semantics is by defining the spatial relationships between the body parts of characters. However, existing methods consider only the character skeleton and thus are not suitable for capturing skin‐level spatial relationships. This paper proposes a novel method for retargeting interaction motions with respect to character skins. Specifically, we introduce the aura mesh, which is a volumetric mesh that surrounds a character's skin. The spatial relationships between two characters are computed from the overlap of the skin mesh of one character and the aura mesh of the other, and then the interaction motion retargeting is achieved by preserving the spatial relationships as much as possible while satisfying other constraints. We show the effectiveness of our method through a number of experiments.     

Sparse Rig Parameter Optimization for Character Animation

We propose a novel motion retargeting method that efficiently estimates artist‐friendly rig space parameters. Inspired by the workflow typically observed in keyframe animation, our approach transfers a source motion into a production friendly character rig by optimizing the rig space parameters while balancing the considerations of fidelity to the source motion and the ease of subsequent editing. We propose the use of an intermediate object to transfer both the skeletal motion and the mesh deformation. The target rig‐space parameters are then optimized to minimize the error between the motion of an intermediate object and the target character. The optimization uses a set of artist defined weights to modulate the effect of the different rig space parameters over time. Sparsity inducing regularizers and keyframe extraction streamline any additional editing processes. The results obtained with different types of character rigs demonstrate the versatility of our method and its effectiveness in simplifying any necessary manual editing within the production pipeline.     

Automatic Registration for Articulated Shapes

We present an unsupervised algorithm for aligning a pair of shapes in the presence of significant articulated motion and missing data, while assuming no knowledge of a template, user‐placed markers, segmentation, or the skeletal structure of the shape. We explicitly sample the motion, which gives a priori the set of possible rigid transformations between parts of the shapes. This transforms the problem into a discrete labeling problem, where the goal is to find an optimal assignment of transformations for aligning the shapes. We then apply graph cuts to optimize a novel cost function, which encodes a preference for a consistent motion assignment from both source to target and target to source. We demonstrate the robustness of our method by aligning several synthetic and real‐world datasets.     

Multi‐View Stereo on Consistent Face Topology

We present a multi‐view stereo reconstruction technique that directly produces a complete high‐fidelity head model with consistent facial mesh topology. While existing techniques decouple shape estimation and facial tracking, our framework jointly optimizes for stereo constraints and consistent mesh parameterization. Our method is therefore free from drift and fully parallelizable for dynamic facial performance capture. We produce highly detailed facial geometries with artist‐quality UV parameterization, including secondary elements such as eyeballs, mouth pockets, nostrils, and the back of the head. Our approach consists of deforming a common template model to match multi‐view input images of the subject, while satisfying cross‐view, cross‐subject, and cross‐pose consistencies using a combination of 2D landmark detection, optical flow, and surface and volumetric Laplacian regularization. Since the flow is never computed between frames, our method is trivially parallelized by processing each frame independently. Accurate rigid head pose is extracted using a PCA‐based dimension reduction and denoising scheme. We demonstrate high‐fidelity performance capture results with challenging head motion and complex facial expressions around eye and mouth regions. While the quality of our results is on par with the current state‐of‐the‐art, our approach can be fully parallelized, does not suffer from drift, and produces face models with production‐quality mesh topologies.     

Reconstructing Animated Meshes from Time‐Varying Point Clouds

In this paper, we describe a novel approach for the reconstruction of animated meshes from a series of time‐deforming point clouds. Given a set of unordered point clouds that have been captured by a fast 3‐D scanner, our algorithm is able to compute coherent meshes which approximate the input data at arbitrary time instances. Our method is based on the computation of an implicit function in ?⁴ that approximates the time‐space surface of the time‐varying point cloud. We then use the four‐dimensional implicit function to reconstruct a polygonal model for the first time‐step. By sliding this template mesh along the time‐space surface in an as‐rigid‐as‐possible manner, we obtain reconstructions for further time‐steps which have the same connectivity as the previously extracted mesh while recovering rigid motion exactly. The resulting animated meshes allow accurate motion tracking of arbitrary points and are well suited for animation compression. We demonstrate the qualities of the proposed method by applying it to several data sets acquired by real‐time 3‐D scanners.     

Hierarchical Deformation of Locally Rigid Meshes

We propose a method for calculating deformations of models by deforming a low‐resolution mesh and adding details while ensuring that the details we add satisfy a set of constraints. Our method builds a low‐resolution representation of a mesh by using edge collapses and performs an as‐rigid‐as‐possible deformation on the simplified mesh. We then add back details by reversing edge‐collapses so that the shape of the mesh is locally preserved. While adding details, we deform the mesh to match the predicted positions of constraints so that constraints on the full‐resolution mesh are met. Our method operates on meshes with arbitrary triangulations, satisfies constraints over the full‐resolution mesh and converges quickly.     

As‐Conformal‐As‐Possible Surface Registration

We present a non‐rigid surface registration technique that can align surfaces with sizes and shapes that are different from each other, while avoiding mesh distortions during deformation. The registration is constrained locally as conformal as possible such that the angles of triangle meshes are preserved, yet local scales are allowed to change. Based on our conformal registration technique, we devise an automatic registration and interactive registration technique, which can reduce user interventions during template fitting. We demonstrate the versatility of our technique on a wide range of surfaces.     

Isometry‐Aware Preconditioning for Mesh Parameterization

This paper presents a new preconditioning technique for large‐scale geometric optimization problems, inspired by applications in mesh parameterization. Our positive (semi‐)definite preconditioner acts on the gradients of optimization problems whose variables are positions of the vertices of a triangle mesh in ?² or of a tetrahedral mesh in ?³, converting localized distortion gradients into the velocity of a globally near‐rigid motion via a linear solve. We pose our preconditioning tool in terms of the Killing energy of a deformation field and provide new efficient formulas for constructing Killing operators on triangle and tetrahedral meshes. We demonstrate that our method is competitive with state‐of‐the‐art algorithms for locally injective parameterization using a variety of optimization objectives and show applications to two‐ and three‐dimensional mesh deformation.     

Deformation‐Driven Shape Correspondence

Non‐rigid 3D shape correspondence is a fundamental and difficult problem. Most applications which require a correspondence rely on manually selected markers. Without user assistance, the performances of existing automatic correspondence methods depend strongly on a good initial shape alignment or shape prior, and they generally do not tolerate large shape variations. We present an automatic feature correspondence algorithm capable of handling large, non‐rigid shape variations, as well as partial matching. This is made possible by leveraging the power of state‐of‐the‐art mesh deformation techniques and relying on a combinatorial tree traversal for correspondence search. The search is deformation‐driven, prioritized by a self‐distortion energy measured on meshes deformed according to a given correspondence. We demonstrate the ability of our approach to naturally match shapes which differ in pose, local scale, part decomposition, and geometric detail through numerous examples.     

Mesh Sequence Morphing

Morphing is an important technique for the generation of special effects in computer animation. However, an analogous technique has not yet been applied to the increasingly prevalent animation representation, i.e. 3D mesh sequences. In this paper, a technique for morphing between two mesh sequences is proposed to simultaneously blend motions and interpolate shapes. Based on all possible combinations of the motions and geometries, a universal framework is proposed to recreate various plausible mesh sequences. To enable a universal framework, we design a skeleton‐driven cage‐based deformation transfer scheme which can account for motion blending and geometry interpolation. To establish one‐to‐one correspondence for interpolating between two mesh sequences, a hybrid cross‐parameterization scheme that fully utilizes the skeleton‐driven cage control structure and adapts user‐specified joint‐like markers, is introduced. The experimental results demonstrate that the framework, not only accomplishes mesh sequence morphing, but also is suitable for a wide range of applications such as deformation transfer, motion blending or transition and dynamic shape interpolation.     

Semi‐Automatic Conversion of 3D Shape into Flat‐Foldable Polygonal Model

This paper presents a method that can convert a given 3D mesh into a flat‐foldable model consisting of rigid panels. A previous work proposed a method to assist manual design of a single component of such flat‐foldable model, consisting of vertically‐connected side panels as well as horizontal top and bottom panels. Our method semi‐automatically generates a more complicated model that approximates the input mesh with multiple convex components. The user specifies the folding direction of each convex component and the fidelity of shape approximation. Given the user inputs, our method optimizes shapes and positions of panels of each convex component in order to make the whole model flat‐foldable. The user can check a folding animation of the output model. We demonstrate the effectiveness of our method by fabricating physical paper prototypes of flat‐foldable models.     

Online Motion Capture Marker Labeling for Multiple Interacting Articulated Targets

In this paper, we propose an online motion capture marker labeling approach for multiple interacting articulated targets. Given hundreds of unlabeled motion capture markers from multiple articulated targets that are interacting each other, our approach automatically labels these markers frame by frame, by fitting rigid bodies and exploiting trained structure and motion models. Advantages of our approach include: 1) our method is an online algorithm, which requires no user interaction once the algorithm starts. 2) Our method is more robust than traditional the closest point-based approaches by automatically imposing the structure and motion models. 3) Due to the use of the structure model which encodes the rigidity of each articulated body of captured targets, our method can recover missing markers robustly. Our approach is efficient and particularly suited for online computer animation and video game applications.     

Pose partitioning for multi‐resolution segmentation of arbitrary mesh animations

We present a complete approach to efficiently deriving a varying level‐of‐detail segmentation of arbitrary animated objects. An over‐segmentation is built by combining sets of initial segments computed for each input pose, followed by a fast progressive simplification which aims at preserving rigid segments. The final segmentation result can be efficiently adjusted for cases where pose editing is performed or new poses are added at arbitrary positions in the mesh animation sequence. A smooth view of pose‐to‐pose segmentation transitions is offered by merging the partitioning of the current pose with that of the next pose. A perceptually friendly visualization scheme is also introduced for propagating segment colors between consecutive poses. We report on the efficiency and quality of our framework as compared to previous methods under a variety of skeletal and highly deformable mesh animations.     

Hexahedral Meshing With Varying Element Sizes

Hexahedral (or Hex‐) meshes are preferred in a number of scientific and engineering simulations and analyses due to their desired numerical properties. Recent state‐of‐the‐art techniques can generate high‐quality hex‐meshes. However, they typically produce hex‐meshes with uniform element sizes and thus may fail to preserve small‐scale features on the boundary surface. In this work, we present a new framework that enables users to generate hex‐meshes with varying element sizes so that small features will be filled with smaller and denser elements, while the transition from smaller elements to larger ones is smooth, compared to the octree‐based approach. This is achieved by first detecting regions of interest (ROIs) of small‐scale features. These ROIs are then magnified using the as‐rigid‐as‐possible deformation with either an automatically determined or a user‐specified scale factor. A hex‐mesh is then generated from the deformed mesh using existing approaches that produce hex‐meshes with uniform‐sized elements. This initial hex‐mesh is then mapped back to the original volume before magnification to adjust the element sizes in those ROIs. We have applied this framework to a variety of man‐made and natural models to demonstrate its effectiveness.     

Animation‐Aware Quadrangulation

Geometric meshes that model animated characters must be designed while taking into account the deformations that the shape will undergo during animation. We analyze an input sequence of meshes with point‐to‐point correspondence, and we automatically produce a quadrangular mesh that fits well the input animation. We first analyze the local deformation that the surface undergoes at each point, and we initialize a cross field that remains as aligned as possible to the principal directions of deformation throughout the sequence. We then smooth this cross field based on an energy that uses a weighted combination of the initial field and the local amount of stretch. Finally, we compute a field‐aligned quadrangulation with an off‐the‐shelf method. Our technique is fast and very simple to implement, and it significantly improves the quality of the output quad mesh and its suitability for character animation, compared to creating the quad mesh based on a single pose. We present experimental results and comparisons with a state‐of‐the‐art quadrangulation method, on both sequences from 3D scanning and synthetic sequences obtained by a rough animation of a triangulated model.     

Mesh Decomposition with Cross‐Boundary Brushes

We present a new intuitive UI, which we call cross‐boundary brushes, for interactive mesh decomposition. The user roughly draws one or more strokes across a desired cut and our system automatically returns a best cut running through all the strokes. By the different natures of part components (i.e., semantic parts) and patch components (i.e., flatter surface patches) in general models, we design two corresponding brushes: part‐brush and patch‐brush. These two types of brushes share a common user interface, enabling easy switch between them. The part‐brush executes a cut along an isoline of a harmonic field driven by the user‐specified strokes. We show that the inherent smoothness of the harmonic field together with a carefully designed isoline selection scheme lead to segmentation results that are insensitive to noise, pose, tessellation and variation in user's strokes. Our patch‐brush uses a novel facet‐based surface metric that alleviates sensitivity to noise and fine details common in region‐growing algorithms. Extensive experimental results demonstrate that our cutting tools can produce user‐desired segmentations for a wide variety of models even with single strokes. We also show that our tools outperform the state‐of‐art interactive segmentation tools in terms of ease of use and segmentation quality.     

Placement of Deformable Objects

With the increasing complexity of photorealistic scenes, the question of building and placing objects in three‐dimensional scenes is becoming ever more difficult. While the question of placement of rigid objects has captured the attention of researchers in the past, this work presents an intuitive and interactive scheme to properly place deformable objects with the aid of free‐form deformation tools. The presented scheme can also be used to animate the locomotion of nonrigid objects, most noticeably animals, and adapt the motion to arbitrary terrain. The automatic construction of our free‐form deformation tool is completely hidden from the end user, and hence, circumvents the difficulties typically faced in manipulating these deformation functions. Further, a precise bound on the error that is introduced by applying free‐form deformations to polygonal models is presented, along with an almost‐optimal adaptive refinement algorithm to achieve a certain accuracy in the mapping.     

ARAPLBS: Robust and Efficient Elasticity‐Based Optimization of Weights and Skeleton Joints for Linear Blend Skinning with Parametrized Bones

We present a fast, robust and high‐quality technique to skin a mesh with reference to a skeleton. We consider the space of possible skeleton deformations (based on skeletal constraints, or skeletal animations), and compute skinning weights based on an optimization scheme to obtain as‐rigid‐as‐possible (ARAP) corresponding mesh deformations. We support stretchable‐and‐twistable bones (STBs) and spines by generalizing the ARAP deformations to stretchable deformers. In addition, our approach can optimize joint placements. If wanted, a user can guide and interact with the results, which is facilitated by an interactive feedback, reached via an efficient sparsification scheme. We demonstrate our technique on challenging inputs (STBs and spines, triangle and tetrahedral meshes featuring missing elements, boundaries, self‐intersections or wire edges).     

Divergence‐Free Shape Correspondence by Deformation

We present a novel approach for solving the correspondence problem between a given pair of input shapes with non‐rigid, nearly isometric pose difference. Our method alternates between calculating a deformation field and a sparse correspondence. The deformation field is constructed with a low rank Fourier basis which allows for a compact representation. Furthermore, we restrict the deformation fields to be divergence‐free which makes our morphings volume preserving. This can be used to extract a correspondence between the inputs by deforming one of them along the deformation field using a second order Runge‐Kutta method and resulting in an alignment of the inputs. The advantages of using our basis are that there is no need to discretize the embedding space and the deformation is volume preserving. The optimization of the deformation field is done efficiently using only a subsampling of the orginal shapes but the correspondence can be extracted for any mesh resolution with close to linear increase in runtime. We show 3D correspondence results on several known data sets and examples of natural intermediate shape sequences that appear as a by‐product of our method.     

Smart Scribbles for Sketch Segmentation

We present ‘Smart Scribbles’—a new scribble‐based interface for user‐guided segmentation of digital sketchy drawings. In contrast to previous approaches based on simple selection strategies, Smart Scribbles exploits richer geometric and temporal information, resulting in a more intuitive segmentation interface. We introduce a novel energy minimization formulation in which both geometric and temporal information from digital input devices is used to define stroke‐to‐stroke and scribble‐to‐stroke relationships. Although the minimization of this energy is, in general, an NP‐hard problem, we use a simple heuristic that leads to a good approximation and permits an interactive system able to produce accurate labellings even for cluttered sketchy drawings. We demonstrate the power of our technique in several practical scenarios such as sketch editing, as‐rigid‐as‐possible deformation and registration, and on‐the‐fly labelling based on pre‐classified guidelines.