Learned Models for Estimation of Rigid and Articulated Human Motion from Stationary or Moving Camera

We propose an approach for modeling, measurement and tracking of rigid and articulated motion as viewed from a stationary or moving camera. We first propose an approach for learning temporal-flow models from exemplar image sequences. The temporal-flow models are represented as a set of orthogonal temporal-flow bases that are learned using principal component analysis of instantaneous flow measurements. Spatial constraints on the temporal-flow are then incorporated to model the movement of regions of rigid or articulated objects. These spatio-temporal flow models are subsequently used as the basis for simultaneous measurement and tracking of brightness motion in image sequences. Then we address the problem of estimating composite independent object and camera image motions. We employ the spatio-temporal flow models learned through observing typical movements of the object from a stationary camera to decompose image motion into independent object and camera motions. The performance of the algorithms is demonstrated on several long image sequences of rigid and articulated bodies in motion.     

Object recognition by combining paraperspective images

This paper provides a study on object recognition under paraperspective projection. Discussed is the problem of determining whether or not a given image was obtained from a 3-D object to be recognized. First it is clarified that paraperspective projection is the first-order approximation of perspective projection. Then it is shown that, if we represent an object as a set of its feature points and the object undergoes a rigid transformation or an affine transformation, any paraperspective image can be expressed as a linear combination of several appropriate paraperspective images: we need at least three images for rigid transformations; whereas we need at least two images for affine transformations. Particularly in the case of a rigid transformation, the coefficients of the combination have to satisfy two conditions: orthogonality and norm equality. A simple algorithm to solve the above problem based on these properties is presented: a linear, single-shot algorithm. Some experimental results with synthetic images and real images are also given.This work was done while the author was with ATR Auditory and Visual Perception Research Laboratories.Advanced Research Laboratory Hitachi, Ltd.     

On differential invariants of planar curves and recognizing partially occluded planar shapes

Viewing transformations like similarity, affine and projective maps may distort planar shapes considerably. However, it is possible to associate local invariant signature functions to smooth boundaries that enable recognition of distorted shapes even in the case of partial occlusion. The derivation of signature functions, generalizing the intrinsic curvature versus arc-length representation in the case of rigid motions in the plane, is based on differential invariants associated to viewing transformation.     

Reconstructing 3D trajectories of independently moving objects using generic constraints

The 3D reconstruction of scenes containing independently moving objects from uncalibrated monocular sequences still poses serious challenges. Even if the background and the moving objects are rigid, each reconstruction is only known up to a certain scale, which results in a one-parameter family of possible, relative trajectories per moving object with respect to the background. In order to determine a realistic solution from this family of possible trajectories, this paper proposes to exploit the increased linear coupling between camera and object translations that tends to appear at false scales. An independence criterion is formulated in the sense of true object and camera motions being minimally correlated. The increased coupling at false scales can also lead to the destruction of special properties such as planarity, periodicity, etc. of the true object motion. This provides us with a second, ‘non-accidentalness’ criterion for the selection of the correct motion among the one-parameter family.     

Model-based invariants for 3-D vision

Invariance under a group of 3-D transformations seems a desirable component of an efficient 3-D shape representation. We propose representations which are invariant under weak perspective to either rigid or linear 3-D transformations, and we show how they can be computed efficiently from a sequence of images with a linear and incremental algorithm. We show simulated results with perspective projection and noise, and the results of model acquisition from a real sequence of images. The use of linear computation, together with the integration through time of invariant representations, offers improved robustness and stability. Using these invariant representations, we derive model-based projective invariant functions of general 3-D objects. We discuss the use of the model-based invariants with existing recognition strategies: alignment without transformation, and constant time indexing from 2-D images of general 3-D objects.     

Computing the generalized aspect graph for objects with movingparts

Algorithms for computing the aspect graph representation are generalized to include a larger, more realistic domain of objects known as articulated assemblies those objects composed of rigid parts with articulated connections allowed between parts. The generalization suggests two slightly different representations: one that directly summarizes the possible general views of the object and another (hierarchical) form summarizing the possible general configurations and their respective views. Algorithms are outlined for computing both representations. The generalized aspect graphs of assemblies formed using translational connections are examined     

Implicit Moment Invariants

The use of traditional moment invariants in object recognition is limited to simple geometric transforms, such as rotation, scaling and affine transformation of the image. This paper introduces so-called implicit moment invariants. Implicit invariants measure the similarity between two images factorized by admissible image deformations. For many types of image deformations traditional invariants do not exist but implicit invariants can be used as features for object recognition. In the paper we present implicit moment invariants with respect to polynomial transform of spatial coordinates, describe their stable and efficient implementation by means of orthogonal moments, and demonstrate their performance in artificial as well as real experiments.     

用二维点对应数据分割和估计多刚体运动

已知含有多个三维刚体的场景,在运动前后的二维点对应数据集合,其中可以包含高斯噪声和出格点数据,发展了初始部分匹配的生成-生长技术并运用刚性约束.将上述二维点对应数据集合.分割成多个分别对应于不同刚体运动的二维点对应数据子集,并能分离出所有出格点数据.再利用单刚体运动估计算法就可估计出各个刚体运动参数.实验结果表明了算法的有效性.     

Model-based recognition of 3D objects from single images

In this work, we treat major problems of object recognition which have received relatively little attention lately. Among them are the loss of depth information in the projection from a 3D object to a single 2D image, and the complexity of finding feature correspondences between images. We use geometric invariants to reduce the complexity of these problems. There are no geometric invariants of a projection from 3D to 2D. However, given certain modeling assumptions about the 3D object, such invariants can be found. The modeling assumptions can be either a particular model or a generic assumption about a class of models. Here, we use such assumptions for single-view recognition. We find algebraic relations between the invariants of a 3D model and those of its 2D image under general projective projection. These relations can be described geometrically as invariant models in a 3D invariant space, illuminated by invariant “light rays,” and projected onto an invariant version of the given image. We apply the method to real images     

On multiple moving objects

This paper explores the motion-planning problem for multiple moving objects. The approach taken consists of assigning priorities to the objects, then planning motions one object at a time. For each moving object, the planner constructs a configuration space-time that represents the time-varying constraints imposed on the moving object by the other moving and stationary objects. The planner represents this space-time approximately, using two-dimensional slices. The space-time is then searched for a collision-free path. The paper demonstrates this approach in two domains. One domain consists of translating planar objects; the other domain consists of two-link planar articulated arms.This report describes research performed at the Artificial Intelligence Laboratory of the Massachusetts Institute of Technology. Michael Erdmann is supported in part by a fellowship from General Motors Research Laboratories. Tomás Lozano-Pérez is supported by an NSF Presidential Young Investigator grant. Support for the Laboratory's Artificial Intelligence research is provided in part by the System Development Foundation, in part by the Office of Naval Research under Office of Naval Research Contract N00014-81-K-0494, and in part by the Advanced Research Projects Agency under Office of Naval Research Contracts N00014-80-C-0505 and N00014-82-K-0344.     

有环境约束时完成一族跟踪任务的力控制

本文为工业机器人提出了一种极点配置控制法，这种控制方法的优点有：一是它的积分作用消除了机器人的微小扰动和稳态误差；二是能任意设置系统的极点，因此能保证闭环系统的稳定性和规定状态变量的暂态响应；三是加入了加速度反馈，抑制了由电枢电感所引起的机械手的振动，最后，给出了ＰＵＭＡ５６２机器人的计算机仿真和实验结果验证了此控制法的有效性。     

3D–2D Projective Registration of Free-Form Curves and Surfaces

Some medical interventions require knowing the correspondence between an MRI/CT image and the actual position of the patient. Examples occur in neurosurgery and radiotherapy, but also in video surgery (laparoscopy). We present in this paper three new techniques for performing this task without artificial markers. To do this, we find the 3D–2D projective transformation (composition of a rigid displacement and a perspective projection) which maps a 3D object onto a 2D image of this object. Depending on the object model (curve or surface), and on the 2D image acquisition system (X-Ray, video), the techniques are different but the framework is common:Results are presented on a variety of real medical data to demonstrate the validity of our approach.     

Quasi-Invariant Parameterisations and Matching of Curves in Images

In this paper, we investigate quasi-invariance on a smooth manifold, and show that there exist quasi-invariant parameterisations which are not exactly invariant but approximately invariant under group transformations and do not require high order derivatives. The affine quasi-invariant parameterisation is investigated in more detail and exploited for defining general affine semi-local invariants from second order derivatives only. The new invariants are implemented and used for matching curve segments under general affine motions and extracting symmetry axes of objects with 3D bilateral symmetry.     

View variation of point-set and line-segment features

The variation, with respect to view, of 2D features defined for projections of 3D point sets and line segments is studied. It is established that general-case view-invariants do not exist for any number of points, given true perspective, weak perspective, or orthographic projection models. Feature variation under the weak perspective approximation is then addressed. Though there are no general-case weak-perspective invariants, there are special-case invariants of practical importance. Those cited in the literature are derived from linear dependence relations and the invariance of this type of relation to linear transformation. The variation with respect to view is studied for an important set of 2D line segment features: the relative orientation, size, and position of one line segment with respect to another. The analysis includes an evaluation criterion for feature utility in terms of view-variation. This relationship is a function of both the feature and the particular configuration of 3D line segments. The use of this information in objection recognition is demonstrated for difficult discrimination tasks     

The analytic structure of image flows: Deformation and segmentation

Time-varying imagery is often described in terms of image flow fields (i.e., image motion), which correspond to the perceptive projection of feature motions in three dimensions (3D). In the case of multiple moving objects with smooth surfaces, the image flow possesses an analytic structure that reflects these 3D properties. This paper describes the analytic structure of image flow fields in the image space-time domain, and its use for segmentation and 3D motion computation. First we discuss thelocal flow structure as embodied in the concept ofneighborhood deformation. The local image deformation is effectively represented by a set of 12 basis deformations, each of which is responsible for an independent deformation. This local representation provides us with sufficient information for the recovery of 3D object structure and motion, in the case of relative rigid body motions. We next discuss theglobal flow structure embodied in the partitioning of the entire image plane intoanalytic regions separated byboundaries of analyticity, such that each small neighborhood within the analytic region is described in terms of deformation bases. This analysis reveals an effective mechanism for detecting the analytic boundaries of flow fields, thereby segmenting the image into meaningful regions. The notion ofconsistency which is often used in the image segmentation is made explicit by the mathematical notion ofanalyticity derived from the projection relation of 3D object motion. The concept of flow analyticity is then extended to the temporal domain, suggesting a more robust algorithm for recovering image flow from multiple frames. Finally, we argue that the process of flow segmentation can be understood in the framework of grouping process. The general concept ofcoherence orgrouping through local support (such as the second-order flows in our case) is discussed.     

Reconstruction Based Recognition of Scenes with Multiple Repeated Components

This paper proposes an invariance based recognition scheme for scenes with multiple repeated components. The scheme considers three component subsets which characterize the scene completely. Each such three component subset is reconstructed using single image based information. We have developed a mathematical framework for the projective reconstruction based on relative affine structure of each such three component building block. This is extended to the case when each of the components is a quadric. A set of projective invariants of three quadrics has also been obtained by us. Although the reconstruction scheme is general and applicable to all multiple repeated components, it requires the computation of infinite homography. The infinite homography and hence the reconstruction scheme are only image computable with the given information in the case of translational repetition. We therefore develop a recognition strategy for the specific case of translationally repeated quadrics. As a recognition strategy for scenes with multiple translationally repeated quadric components, we propose to compute and store invariant values for each such three component subsets. Experiments on real data have shown the applicability of this approach for recognition of aerial images of power plants. The discriminatory power of the invariants and the stability of the recognition results have also been experimentally demonstrated.     

Fluid Simulation with Articulated Bodies

We present an algorithm for creating realistic animations of characters that are swimming through fluids. Our approach combines dynamic simulation with data-driven kinematic motions (motion capture data) to produce realistic animation in a fluid. The interaction of the articulated body with the fluid is performed by incorporating joint constraints with rigid animation and by extending a solid/fluid coupling method to handle articulated chains. Our solver takes as input the current state of the simulation and calculates the angular and linear accelerations of the connected bodies needed to match a particular motion sequence for the articulated body. These accelerations are used to estimate the forces and torques that are then applied to each joint. Based on this approach, we demonstrate simulated swimming results for a variety of different strokes, including crawl, backstroke, breaststroke, and butterfly. The ability to have articulated bodies interact with fluids also allows us to generate simulations of simple water creatures that are driven by simple controllers.     

Target Detection Based on 3D Multi-Component Model and Inverse Projection Transformation

Target detection based on image/video, being involved to deal with the geometry and scale deformation, as well as the change in the form of movement caused by camera imaging, algorithms are always designed complexly. Though, object shelter and adhesion still cannot be well resolved. Considering of that, a new method for target detection on true 3D space based on the inverse projection transformation and a mixing component model is proposed. Firstly, the inverse projective arrays parallel to target local surface are established on 3D space. Then, the 2D image is inversely projected to these planes through 3D point cloud re-projection, and a lot of inverse projective images with target local apparent characteristics are gained. After that, component HOG feature dictionaries are trained using the inverse projective images as samples, and on account of it, sparse decomposition approach is adopted to detect target local components. Finally, 3D centroid clustering for all the components is further used to identify the target. Experiment results indicate that the target detection method on true 3D space based on multi-components model and inverse projection transformation can not only deal with the object occlusion and adhesion perfectly, but also adapt to the multi-angle target detection well, and the accuracy and speed is far beyond that of the algorithm on 2D image.     

Tracking and Motion Estimation of the Articulated Object: a Hierarchical Kalman Filter Approach

Real-time motion capture plays a very important role in various applications, such as 3D interface for virtual reality systems, digital puppetry, and real-time character animation. In this paper we challenge the problem of estimating and recognizing the motion of articulated objects using theoptical motion capturetechnique. In addition, we present an effective method to control the articulated human figure in realtime.The heart of this problem is the estimation of 3D motion and posture of an articulated, volumetric object using feature points from a sequence of multiple perspective views. Under some moderate assumptions such as smooth motion and known initial posture, we develop a model-based technique for the recovery of the 3D location and motion of a rigid object using a variation of Kalman filter. The posture of the 3D volumatric model is updated by the 2D image flow of the feature points for all views. Two novel concepts – the hierarchical Kalman filter (KHF) and the adaptive hierarchical structure (AHS) incorporating the kinematic properties of the articulated object – are proposed to extend our formulation for the rigid object to the articulated one. Our formulation also allows us to avoid two classic problems in 3D tracking: the multi-view correspondence problem, and the occlusion problem. By adding more cameras and placing them appropriately, our approach can deal with the motion of the object in a very wide area. Furthermore, multiple objects can be handled by managing multiple AHSs and processing multiple HKFs.We show the validity of our approach using the synthetic data acquired simultaneously from the multiple virtual camera in a virtual environment (VE) and real data derived from a moving light display with walking motion. The results confirm that the model-based algorithm works well on the tracking of multiple rigid objects.     

A simple approximation to rigid body dynamics for computer animation

Recently, the dynamics of linked articulated rigid bodies has become a valuable tool for making realistic three-dimensional computer animations. An exact treatment of rigid body dynamics, however, is based on rather non-intuitive results from classical mechanics (e.g. the Euler equations for rotating bodies) and it relies heavily on sophisticated numerical schemes to solve (large) sets of coupled non-linear algebraic and differential equations. As a result, articulated rigid bodies are not yet supported by most real-time animation systems. This paper discusses an approach to rigid body dynamics which is based on (both conceptually and algorithmically much simpler) point mechanics; this gives rise to an asymptotically exact numerical scheme (NSI) which is useful in the context of real-time animation, provided that the number of degrees of freedom of the simulated system is not too large. Based on NSI, a second scheme (NS2) is derived which is useful for approximating the motions of linked articulated rigid bodies; NS2 turns out to be sufficiently fast to give at least qualitative results in real-time simulation. In general, the algorithm NS2 is not necessarily (asymptotically) exact, but a quantitative analysis shows that in the absence of reaction forces it conserves angular momentum.