Interactive motion mapping for real‐time character control

It is now possible to capture the 3D motion of the human body on consumer hardware and to puppet in real time skeleton‐based virtual characters. However, many characters do not have humanoid skeletons. Characters such as spiders and caterpillars do not have boned skeletons at all, and these characters have very different shapes and motions. In general, character control under arbitrary shape and motion transformations is unsolved ‐ how might these motions be mapped? We control characters with a method which avoids the rigging‐skinning pipeline — source and target characters do not have skeletons or rigs. We use interactively‐defined sparse pose correspondences to learn a mapping between arbitrary 3D point source sequences and mesh target sequences. Then, we puppet the target character in real time. We demonstrate the versatility of our method through results on diverse virtual characters with different input motion controllers. Our method provides a fast, flexible, and intuitive interface for arbitrary motion mapping which provides new ways to control characters for real‐time animation.     

Temporally Consistent Motion Segmentation From RGB‐D Video

Temporally consistent motion segmentation from RGB‐D videos is challenging because of the limitations of current RGB‐D sensors. We formulate segmentation as a motion assignment problem, where a motion is a sequence of rigid transformations through all frames of the input. We capture the quality of each potential assignment by defining an appropriate energy function that accounts for occlusions and a sensor‐specific noise model. To make energy minimization tractable, we work with a discrete set instead of the continuous, high dimensional space of motions, where the discrete motion set provides an upper bound for the original energy. We repeatedly minimize our energy, and in each step extend and refine the motion set to further lower the bound. A quantitative comparison to the current state of the art demonstrates the benefits of our approach in difficult scenarios.     

Deformable 3D Shape Registration Based on Local Similarity Transforms

In this paper, a new method for deformable 3D shape registration is proposed. The algorithm computes shape transitions based on local similarity transforms which allows to model not only as‐rigid‐as‐possible deformations but also local and global scale. We formulate an ordinary differential equation (ODE) which describes the transition of a source shape towards a target shape. We assume that both shapes are roughly pre‐aligned (e.g., frames of a motion sequence). The ODE consists of two terms. The first one causes the deformation by pulling the source shape points towards corresponding points on the target shape. Initial correspondences are estimated by closest‐point search and then refined by an efficient smoothing scheme. The second term regularizes the deformation by drawing the points towards locally defined rest positions. These are given by the optimal similarity transform which matches the initial (undeformed) neighborhood of a source point to its current (deformed) neighborhood. The proposed ODE allows for a very efficient explicit numerical integration. This avoids the repeated solution of large linear systems usually done when solving the registration problem within general‐purpose non‐linear optimization frameworks. We experimentally validate the proposed method on a variety of real data and perform a comparison with several state‐of‐the‐art approaches.     

Autocorrelation Descriptor for Efficient Co‐Alignment of 3D Shape Collections

Co‐aligning a collection of shapes to a consistent pose is a common problem in shape analysis with applications in shape matching, retrieval and visualization. We observe that resolving among some orientations is easier than others, for example, a common mistake for bicycles is to align front‐to‐back, while even the simplest algorithm would not erroneously pick orthogonal alignment. The key idea of our work is to analyse rotational autocorrelations of shapes to facilitate shape co‐alignment. In particular, we use such an autocorrelation measure of individual shapes to decide which shape pairs might have well‐matching orientations; and, if so, which configurations are likely to produce better alignments. This significantly prunes the number of alignments to be examined, and leads to an efficient, scalable algorithm that performs comparably to state‐of‐the‐art techniques on benchmark data sets, but requires significantly fewer computations, resulting in 2–16× speed improvement in our tests.     

Analogy‐driven 3D style transfer

Style transfer aims to apply the style of an exemplar model to a target one, while retaining the target's structure. The main challenge in this process is to algorithmically distinguish style from structure, a high‐level, potentially ill‐posed cognitive task. Inspired by cognitive science research we recast style transfer in terms of shape analogies. In IQ testing, shape analogy queries present the subject with three shapes: source, target and exemplar, and ask them to select an output such that the transformation, or analogy, from the exemplar to the output is similar to that from the source to the target. The logical process involved in identifying the source‐to‐target analogies implicitly detects the structural differences between the source and target and can be used effectively to facilitate style transfer. Since the exemplar has a similar structure to the source, applying the analogy to the exemplar will provide the output we seek. The main technical challenge we address is to compute the source to target analogies, consistent with human logic. We observe that the typical analogies we look for consist of a small set of simple transformations, which when applied to the exemplar generate a continuous, seamless output model. To assemble a shape analogy, we compute an optimal set of source‐to‐target transformations, such that the assembled analogy best fits these criteria. The assembled analogy is then applied to the exemplar shape to produce the desired output model. We use the proposed framework to seamlessly transfer a variety of style properties between 2D and 3D objects and demonstrate significant improvements over the state of the art in style transfer. We further show that our framework can be used to successfully complete partial scans with the help of a user provided structural template, coherently propagating scan style across the completed surfaces.     

CR-Morph: Controllable Rigid Morphing for 2D Animation

This paper presents a rigidity-preserving morphing technique that blends a pair of 2D shapes in a controllable manner. The morphing is controllable in two aspects:1) motion dynamics in the interpolation sequences can be effectively enhanced through an intuitive skeleton control and 2) not only the boundaries but also the interior features of the source and target shapes are precisely aligned during the morphing. We introduce a new compatible triangulation algorithm to decompose the source and target shapes into isomorphic triangles. Moreover, a robust and motion-controllable rigiditypreserving transformation scheme is proposed to blend the compatible triangulations, ultimately leading to a morphing sequence which is appearance-preserving and with the desired motion dynamics. Our approach constitutes a powerful and easy-to-use morphing tool for two-dimensional animation. We demonstrate its versatility, effectiveness and visual accuracy through a variety of examples and comparisons to prior work.     

Learning to Transfer In-Hand Manipulations Using a Greedy Shape Curriculum

In-hand object manipulation is challenging to simulate due to complex contact dynamics, non-repetitive finger gaits, and the need to indirectly control unactuated objects. Further adapting a successful manipulation skill to new objects with different shapes and physical properties is a similarly challenging problem. In this work, we show that natural and robust in-hand manipulation of simple objects in a dynamic simulation can be learned from a high quality motion capture example via deep reinforcement learning with careful designs of the imitation learning problem. We apply our approach on both single-handed and two-handed dexterous manipulations of diverse object shapes and motions. We then demonstrate further adaptation of the example motion to a more complex shape through curriculum learning on intermediate shapes morphed between the source and target object. While a naive curriculum of progressive morphs often falls short, we propose a simple greedy curriculum search algorithm that can successfully apply to a range of objects such as a teapot, bunny, bottle, train, and elephant.     

Brushables: Example‐based Edge‐aware Directional Texture Painting

In this paper we present Brushables—a novel approach to example‐based painting that respects user‐specified shapes at the global level and preserves textural details of the source image at the local level. We formulate the synthesis as a joint optimization problem that simultaneously synthesizes the interior and the boundaries of the region, transferring relevant content from the source to meaningful locations in the target. We also provide an intuitive interface to control both local and global direction of textural details in the synthesized image. A key advantage of our approach is that it enables a “combing” metaphor in which the user can incrementally modify the target direction field to achieve the desired look. Based on this, we implement an interactive texture painting tool capable of handling more complex textures than ever before, and demonstrate its versatility on difficult inputs including vegetation, textiles, hair and painting media.     

Parametric estimation of affine deformations of planar shapes

We consider the estimation of affine transformations aligning a known 2D shape and its distorted observation. The classical way to solve this registration problem is to find correspondences between the shapes and then compute the transformation parameters from these landmarks. Here we propose a novel approach where the exact transformation is obtained as the solution of a polynomial system of equations. The method has been tested on synthetic as well as on real images and its robustness in the presence of segmentation errors and additive geometric noise has also been demonstrated. We have successfully applied the method for the registration of hip prosthesis X-ray images. The advantage of the proposed solution is that it is fast, easy to implement, has linear time complexity, works without established correspondences and provides an exact solution regardless of the magnitude of transformation.     

Improved Approximation Bounds for Planar Point Pattern Matching

We analyze the performance of simple algorithms for matching two planar point sets under rigid transformations so as to minimize the directed Hausdorff distance between the sets. This is a well studied problem in computational geometry. Goodrich, Mitchell, and Orletsky presented a very simple approximation algorithm for this problem, which computes transformations based on aligning pairs of points. They showed that their algorithm achieves an approximation ratio of 4. We introduce a modification to their algorithm, which is based on aligning midpoints rather than endpoints. This modification has the same simplicity and running time as theirs, and we show that it achieves a better approximation ratio of roughly 3.14. We also analyze the approximation ratio in terms of a instance-specific parameter that is based on the ratio between diameter of the pattern set to the optimum Hausdorff distance. We show that as this ratio increases (as is common in practical applications) the approximation ratio approaches 3 in the limit. We also investigate the performance of the algorithm by Goodrich et al. as a function of this ratio, and present nearly matching lower bounds on the approximation ratios of both algorithms. This work was supported by the National Science Foundation under grants CCR-0098151 and CCF-0635099.     

An efficient volumetric method for non-rigid registration

We propose a novel and efficient volumetric method for registering 3D shapes with non-rigid deformations. Our method uses a signed distance field to represent the 3D input shapes and registers them by minimizing the difference between their distance fields. With the assumptions that the sampling points in each cell of the object volume follow the same rigid transformation, and the transformations of the sampling cells vary smoothly inside the object volume, a two-step method is used for the non-rigid registration. The first step is the locally rigid registration, which minimizes the difference between the source and target distance fields of the sampling cells. The second step is the globally non-rigid registration, which minimizes the difference between the transformations of adjacent cells. In just a few iterations, our method rapidly converges for the registration. We tested our method on several datasets, and the experimental results demonstrate the robustness and efficiency of our method.     

Space–time blending

Shape transformation between objects of different topology and positions in space is an open modelling problem. We propose a new approach to solving this problem for two given 2D or 3D shapes. The key steps of the proposed algorithm are: increase dimension by converting two input kD shapes into half‐cylinders in (k+1)D space–time, applying bounded blending with added material to the half‐cylinders, and making cross‐sections for getting intermediate shapes under the transformation. The additional dimension is considered as a time coordinate for making animation. We use the bounded blending set operations in space–time defined using R‐functions and displacement functions with the localized area of influence applied to the functionally defined half‐cylinders. The proposed approach is general enough to handle input shapes with arbitrary topology defined as polygonal objects with holes and disjoint components, set‐theoretic objects, or analytically defined implicit surfaces. The obtained unusual amoeba‐like behaviour of the shape combines metamorphosis with the non‐linear motion. Copyright © 2004 John Wiley & Sons, Ltd.     

Robust Physics‐based Motion Retargeting with Realistic Body Shapes

Motion capture is often retargeted to new, and sometimes drastically different, characters. When the characters take on realistic human shapes, however, we become more sensitive to the motion looking right. This means adapting it to be consistent with the physical constraints imposed by different body shapes. We show how to take realistic 3D human shapes, approximate them using a simplified representation, and animate them so that they move realistically using physically‐based retargeting. We develop a novel spacetime optimization approach that learns and robustly adapts physical controllers to new bodies and constraints. The approach automatically adapts the motion of the mocap subject to the body shape of a target subject. This motion respects the physical properties of the new body and every body shape results in a different and appropriate movement. This makes it easy to create a varied set of motions from a single mocap sequence by simply varying the characters. In an interactive environment, successful retargeting requires adapting the motion to unexpected external forces. We achieve robustness to such forces using a novel LQR‐tree formulation. We show that the simulated motions look appropriate to each character's anatomy and their actions are robust to perturbations.     

Motion planning for tree climbing with inchworm‐like robots

This paper proposes a global path‐ and motion‐planning algorithm that enables inchworm‐like robots to navigate their way up tree branches. The intuitive climbing space representation method proposed here greatly simplifies the path‐planning problem. The dynamic programming algorithm can be used to identify the optimal path leading to the target position in the target direction according to the constraints and requirements specified. The planned path can be applied in any tree‐climbing robot that utilizes the nonenclosure gripping method. An efficient motion‐planning algorithm for continuum inchworm‐like robots is then developed to enable them to climb along the planned path with a high degree of accuracy. In comparison with the method proposed in our previous study, the method proposed herein significantly improves consistency between the planned path and the motions of the robot, and therefore makes it more practical to implement the motion‐planning algorithm in trees of different shapes. The paper also describes hardware experiments in which the proposed planning algorithm is applied to enable inchworm‐like robots to climb real trees, thus validating the proposed planning algorithm in practice. © 2012 Wiley Periodicals, Inc.     

Structured Regularization of Functional Map Computations

We consider the problem of non‐rigid shape matching using the functional map framework. Specifically, we analyze a commonly used approach for regularizing functional maps, which consists in penalizing the failure of the unknown map to commute with the Laplace‐Beltrami operators on the source and target shapes. We show that this approach has certain undesirable fundamental theoretical limitations, and can be undefined even for trivial maps in the smooth setting. Instead we propose a novel, theoretically well‐justified approach for regularizing functional maps, by using the notion of the resolvent of the Laplacian operator. In addition, we provide a natural one‐parameter family of regularizers, that can be easily tuned depending on the expected approximate isometry of the input shape pair. We show on a wide range of shape correspondence scenarios that our novel regularization leads to an improvement in the quality of the estimated functional, and ultimately pointwise correspondences before and after commonly‐used refinement techniques.     

Global Intrinsic Symmetries of Shapes

Although considerable attention in recent years has been given to the problem of symmetry detection in general shapes, few methods have been developed that aim to detect and quantify the intrinsic symmetry of a shape rather than its extrinsic, or pose‐dependent symmetry. In this paper, we present a novel approach for efficiently computing symmetries of a shape which are invariant up to isometry preserving transformations. We show that the intrinsic symmetries of a shape are transformed into the Euclidean symmetries in the signature space defined by the eigenfunctions of the Laplace‐Beltrami operator. Based on this observation, we devise an algorithm which detects and computes the isometric mappings from the shape onto itself. We show that our approach is both computationally efficient and robust with respect to small non‐isometric deformations, even if they include topological changes.     

Glare Generation Based on Wave Optics

This paper proposes a novel and general method of glare generation based on wave optics. A glare image is regarded as a result of Fraunhofer diffraction, which is equivalent to a 2D Fourier transform of the image of given apertures or obstacles. In conventional methods, the shapes of glare images are categorized according to their source apertures, such as pupils and eyelashes and their basic shapes (e.g. halos, coronas, or radial streaks) are manually generated as templates, mainly based on statistical observation. Realistic variations of these basic shapes often depend on the use of random numbers. Our proposed method computes glare images fully automatically from aperture images and can be applied universally to all kinds of apertures, including camera diaphragms. It can handle dynamic changes in the position of the aperture relative to the light source, which enables subtle movement or rotation of glare streaks. Spectra can also be simulated in the glare, since the intensity of diffraction depends on the wavelength of light. The resulting glare image is superimposed onto a given computer‐generated image containing high‐intensity light sources or reflections, aligning the center of the glare image to the high‐intensity areas. Our method is implemented as a multipass rendering software. By precomputing the dynamic glare image set and putting it into texture memory, the software runs at an interactive rate.     

A New Real Time Geometric Transformation Matrix and its Efficient VLSI Implementation

The geometric transformation of an image with irregular shapes from a source space to a target space requires a huge number of multiplications and additions for each pixel. Such a through put is impossible to deliver by any processor in real-time. This paper presents a new transformation matrix which can be used toper form real-time 2-D geometric transformations economically. The implementation of the transformation engine that is used to execute the transformation matrix is also presented. Experimental results indicate that real-time geometric transformations based on a 4 × 4 transformation mesh can be achieved with the use of the transformation engine. This real-time transformation function is useful in warping and blending of images, and can be easily extended to perform irregular shape transformations with a large number of control points in the transformation mesh.