Pose Estimation in Conformal Geometric Algebra Part I: The Stratification of Mathematical Spaces

2D-3D pose estimation means to estimate the relative position and orientation of a 3D object with respect to a reference camera system. This work has its main focus on the theoretical foundations of the 2D-3D pose estimation problem: We discuss the involved mathematical spaces and their interaction within higher order entities. To cope with the pose problem (how to compare 2D projective image features with 3D Euclidean object features), the principle we propose is to reconstruct image features (e.g. points or lines) to one dimensional higher entities (e.g. 3D projection rays or 3D reconstructed planes) and express constraints in the 3D space. It turns out that the stratification hierarchy [11] introduced by Faugeras is involved in the scenario. But since the stratification hierarchy is based on pure point concepts a new algebraic embedding is required when dealing with higher order entities. The conformal geometric algebra (CGA) [24] is well suited to solve this problem, since it subsumes the involved mathematical spaces. Operators are defined to switch entities between the algebras of the conformal space and its Euclidean and projective subspaces. This leads to another interpretation of the stratification hierarchy, which is not restricted to be based solely on point concepts. This work summarizes the theoretical foundations needed to deal with the pose problem. Therefore it contains mainly basics of Euclidean, projective and conformal geometry. Since especially conformal geometry is not well known in computer science, we recapitulate the mathematical concepts in some detail. We believe that this geometric model is useful also for many other computer vision tasks and has been ignored so far. Applications of these foundations are presented in Part II [36].Bodo Rosenhahn gained his diploma degree in Computer Science in 1999. Since then he has been pursuing his Ph.D. at the Cognitive Systems Group, Institute of Computer Science, Christian-Albrechts University Kiel, Germany. He is working on geometric applications of Clifford algebras in computer vision.Prof. Dr. Gerald Sommer received a diploma degree in physics from the Friedrich-Schiller-Universität Jena, Germany, in 1969, a Ph.D. degree in physics from the same university in 1975, and a habilitation degree in engineering from the Technical University Ilmenau, Germany, in 1988. Since 1993 he is leading the research group Cognitive Systems at the Christian-Albrechts-Universität Kiel, Germany. Currently he is also the scientific coordinator of the VISATEC project.     

Pose Estimation in Conformal Geometric Algebra Part II: Real-Time Pose Estimation Using Extended Feature Concepts

Part II uses the foundations of Part I [35] to define constraint equations for 2D-3D pose estimation of different corresponding entities. Most articles on pose estimation concentrate on specific types of correspondences, mostly between points, and only rarely use line correspondences. The first aim of this part is to extend pose estimation scenarios to correspondences of an extended set of geometric entities. In this context we are interested to relate the following (2D) image and (3D) model types: 2D point/3D point, 2D line/3D point, 2D line/3D line, 2D conic/3D circle, 2D conic/3D sphere. Furthermore, to handle articulated objects, we describe kinematic chains in this context in a similar manner. We ensure that all constraint equations end up in a distance measure in the Euclidean space, which is well posed in the context of noisy data. We also discuss the numerical estimation of the pose. We propose to use linearized twist transformations which result in well conditioned and fast solvable systems of equations. The key idea is not to search for the representation of the Lie group, describing the rigid body motion, but for the representation of their generating Lie algebra. This leads to real-time capable algorithms.Bodo Rosenhahn gained his diploma degree in Computer Science in 1999. Since then he has been pursuing his Ph.D. at the Cognitive Systems Group, Institute of Computer Science, Christian-Albrechts University Kiel, Germany. He is working on geometric applications of Clifford algebras in computer vision.Prof. Dr. Gerald Sommer received a diploma degree in physics from the Friedrich-Schiller-Universität Jena, Germany, in 1969, a Ph.D. degree in physics from the same university in 1975, and a habilitation degree in engineering from the Technical University Ilmenau, Germany, in 1988. Since 1993 he is leading the research group Cognitive Systems at the Christian-Albrechts-Universität Kiel, Germany. Currently he is also the scientific coordinator of the VISATEC project.     

Pose Estimation using Point and Line Correspondences

The problem of a real-time pose estimation between a 3D scene and a single camera is a fundamental task in most 3D computer vision and robotics applications such as object tracking, visual servoing, and virtual reality. In this paper we present two fast methods for estimating the 3D pose using 2D to 3D point and line correspondences. The first method is based on the iterative use of a weak perspective camera model and forms a generalization of DeMenthon's method (1995) which consists of determining the pose from point correspondences. In this method the pose is iteratively improved with a weak perspective camera model and at convergence the computed pose corresponds to the perspective camera model. The second method is based on the iterative use of a paraperspective camera model which is a first order approximation of perspective. We describe in detail these two methods for both non-planar and planar objects. Experiments involving synthetic data as well as real range data indicate the feasibility and robustness of these two methods. We analyse the convergence of these methods and we conclude that the iterative paraperspective method has better convergence properties than the iterative weak perspective method. We also introduce a non-linear optimization method for solving the pose problem.     

共形几何代数与运动和形状的刻画

共形几何代数在基于运动和形状刻画的视觉和图形学若干问题中的应用,反映了它能够提供统一和有效的表示和算法,这些应用主要集中在采纳几何体的Grassmann分级表示以及刚体运动的旋量和扭量表示.着重介绍了Grassmann分级表示如何被应用于单眼视觉问题并带来解决方法的简化;通过对刚体运动不同表示的分析,介绍旋量和扭量表示如何克服刚体运动蹬矩阵表示中参数空间具有过多非线性约束的缺点,从而为姿态估计、形状逼近和曲线拼接等问题的解决提供简化方案.     

Integration of Hough Transform of lines and planes in the framework of conformal geometric algebra for 2D and 3D robot vision

This paper presents the application of 2D and 3D Hough Transforms together with conformal geometric algebra to build 3D geometric maps using the geometric entities of lines and planes. Among several existing techniques for robot self-localization, a new approach is proposed for map matching in the Hough domain. The geometric Hough representation is formulated in such a way that one can easily relate it to the conformal geometric algebra framework; thus, the detected lines and planes can be used for algebra-of-incidence computations to find geometric constraints, useful when perceiving special configurations in 3D visual space for exploration, navigation, relocation and obstacle avoidance. We believe that this work is very useful for 2D and 3D geometric pattern recognition in robot vision tasks.     

Geometric techniques for 3D tracking of ultrasound sensor,tumor segmentation in ultrasound images,and 3D reconstruction

This paper presents different methods, some based on geometric algebra, for ultrasound probe tracking in endoscopic images, 3D allocation of the ultrasound probe, ultrasound image segmentation (to extract objects like tumors), and 3D reconstruction of the surface defined by a set of points. The tracking of the ultrasound probe in endoscopic images is done with a particle filter and an auxiliary method based on thresholding in the HSV space. The 3D pose of the ultrasound probe is calculated using conformal geometric algebra (to locate each slide in 3D space). Each slide (ultrasound image) is segmented using two methods: the level-set method and the morphological operators approach in order to obtain the object we are interested in. The points on the object of interest are obtained from the segmented ultrasound images, and then a 3D object is obtained by refining the convex hull. To do that, a peeling process with an adaptive radius is applied, all of this in the geometric algebra framework. Results for points from ultrasound images, as well as for points from objects from the AimatShape Project, are presented (A.I.M.A.T.S.H.A.P.E. – Advanced an Innovative Models And Tools for the development of Semantic-based systems for Handling, Acquiring, and Processing knowledge Embedded in multidimensional digital objects).     

Large-scale multiview 3D hand pose dataset

Accurate visual hand pose estimation at joint level has several applications for human-robot interaction, natural user interfaces and virtual/augmented reality applications. However, it is still an open problem being addressed by the computer vision community. Recent novel deep learning techniques may help circumvent the limitations of standard approaches. However, they require large amounts of accurate annotated data.Hand pose datasets that have been released so far present issues such as limited number of samples, inaccurate data or high-level annotations. Moreover, most of them are focused on depth-based approaches, providing only depth information (missing RGB data).In this work, we present a novel multiview hand pose dataset in which we provide hand color images and different kind of annotations for each sample, i.e. the bounding box and the 2D and 3D location on the joints in the hand. Furthermore, we introduce a simple yet accurate deep learning architecture for real-time robust 2D hand pose estimation. Then, we conduct experiments that show how the use of the proposed dataset in the training stage produces accurate results for 2D hand pose estimation using a single color camera.     

基于多重几何约束的未知物体抓取位姿估计

针对机器人在非结构化环境下面临的未知物体难以快速稳定抓取的问题,提出一种基于多重几何约束的未知物体抓取位姿估计方法.通过深度相机获取场景的几何点云信息,对点云进行预处理得到目标物体,利用简化的夹持器几何形状约束生成抓取位姿样本.然后,利用简化的力封闭约束对样本进行快速粗筛选.对抓取位姿的抓取几何轮廓进行力平衡约束分析,将稳定的位姿传送至机器人执行抓取.采用深度相机与6自由度机械臂组成实验平台,对不同姿态形状的物体进行抓取实验.实验结果表明,本文方法能够有效应对物体种类繁多、缺乏3维模型的情况,在单目标和多目标场景均具有良好的适用性.     

Free-form geometric modeling by integrating parametric and implicit PDEs

Parametric PDE techniques, which use partial differential equations (PDEs) defined over a 2D or 3D parametric domain to model graphical objects and processes, can unify geometric attributes and functional constraints of the models. PDEs can also model implicit shapes defined by level sets of scalar intensity fields. In this paper, we present an approach that integrates parametric and implicit trivariate PDEs to define geometric solid models containing both geometric information and intensity distribution subject to flexible boundary conditions. The integrated formulation of second-order or fourth-order elliptic PDEs permits designers to manipulate PDE objects of complex geometry and/or arbitrary topology through direct sculpting and free-form modeling. We developed a PDE-based geometric modeling system for shape design and manipulation of PDE objects. The integration of implicit PDEs with parametric geometry offers more general and arbitrary shape blending and free-form modeling for objects with intensity attributes than pure geometric models     

Real-time markerless tracking for augmented reality: the virtual visual servoing framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.     

Multi-view 3D Human Pose Estimation in Complex Environment

We introduce a framework for unconstrained 3D human upper body pose estimation from multiple camera views in complex environment. Its main novelty lies in the integration of three components: single-frame pose recovery, temporal integration and model texture adaptation. Single-frame pose recovery consists of a hypothesis generation stage, in which candidate 3D poses are generated, based on probabilistic hierarchical shape matching in each camera view. In the subsequent hypothesis verification stage, the candidate 3D poses are re-projected into the other camera views and ranked according to a multi-view likelihood measure. Temporal integration consists of computing K-best trajectories combining a motion model and observations in a Viterbi-style maximum-likelihood approach. Poses that lie on the best trajectories are used to generate and adapt a texture model, which in turn enriches the shape likelihood measure used for pose recovery. The multiple trajectory hypotheses are used to generate pose predictions, augmenting the 3D pose candidates generated at the next time step.     

A systematic approach for 2D-image to 3D-range registration in urban environments

The photorealistic modeling of large-scale objects, such as urban scenes, requires the combination of range sensing technology and digital photography. In this paper, we attack the key problem of camera pose estimation, in an automatic and efficient way. First, the camera orientation is recovered by matching vanishing points (extracted from 2D images) with 3D directions (derived from a 3D range model). Then, a hypothesis-and-test algorithm computes the camera positions with respect to the 3D range model by matching corresponding 2D and 3D linear features. The camera positions are further optimized by minimizing a line-to-line distance. The advantage of our method over earlier work has to do with the fact that we do not need to rely on extracted planar facades, or other higher-order features; we are utilizing low-level linear features. That makes this method more general, robust, and efficient. We have also developed a user-interface for allowing users to accurately texture-map 2D images onto 3D range models at interactive rates. We have tested our system in a large variety of urban scenes.     

Toward coherent object detection and scene layout understanding

Detecting objects in complex scenes while recovering the scene layout is a critical functionality in many vision-based applications. In this work, we advocate the importance of geometric contextual reasoning for object recognition. We start from the intuition that objects' location and pose in the 3D space are not arbitrarily distributed but rather constrained by the fact that objects must lie on one or multiple supporting surfaces. We model such supporting surfaces by means of hidden parameters (i.e. not explicitly observed) and formulate the problem of joint scene reconstruction and object recognition as the one of finding the set of parameters that maximizes the joint probability of having a number of detected objects on K supporting planes given the observations. As a key ingredient for solving this optimization problem, we have demonstrated a novel relationship between object location and pose in the image, and the scene layout parameters (i.e. normal of one or more supporting planes in 3D and camera pose, location and focal length). Using a novel probabilistic formulation and the above relationship our method has the unique ability to jointly: i) reduce false alarm and false negative object detection rate; ii) recover object location and supporting planes within the 3D camera reference system; iii) infer camera parameters (view point and the focal length) from just one single uncalibrated image. Quantitative and qualitative experimental evaluation on two datasets (desk-top dataset [1] and LabelMe [2]) demonstrates our theoretical claims.     

Articulated body motion tracking using illumination invariant optical flow

We propose a model-based tracking method for articulated objects in monocular video sequences under varying illumination conditions. The tracking method uses estimates of optical flows constructed by projecting model textures into the camera images and comparing the projected textures with the recorded information. An articulated body is modelled in terms of 3D primitives, each possessing a specified texture on its surface. An important step in model-based tracking of 3D objects is the estimation of the pose of the object during the tracking process. The optimal pose is estimated by minimizing errors between the computed optical flow and the projected 2D velocities of the model textures. This estimation uses a least-squares method with kinematic constraints for the articulated object and a perspective camera model. We test our framework with an articulated robot and show results.     

Hierarchical topology based hand pose estimation from a single depth image

Hand pose estimation benefits large human computer interaction applications. The hand pose has high dimensions of freedom (dof) for joints, and various hand poses are flexible. Hand pose estimation is still a challenge problem. Since hand joints on the hand skeleton topology model have strict relationships between each other, we propose a hierarchical topology based approach to estimate 3D hand poses. First, we determine palm positions and palm orientations by detecting hand fingertips and calculating their directions in depth images. It is the global topology of hand poses. Moreover, we define connection relationships of finger joints as the local topology of hand model. Based on hierarchical topology, we extract angle features to describe hand poses, and adopt the regression forest algorithm to estimate 3D coordinates of hand joints. We further use freedom forrest algorithm to refine ambiguous poses in estimation to solve error accumulation problem. The hierarchical topology based approach ensures estimated hand poses in a reasonable topology, and improves estimation accuracy. We evaluate our approach on two public databases, and experiments illustrate its efficiency. Compared with state-of-the-art approaches, our approach improves estimation accuracy.     

Pose Estimation for 3D Workpiece Grasping in Industrial Environment Based on Evolutionary Algorithm

In industrial fields, precise pose of a 3D workpiece can guide operations like grasping and assembly tasks, thus precise estimation of pose of a 3D workpiece has received intensive attention over the last decades. When utilizing vision system as the source of pose estimation, it is difficult to get the pose of a 3D workpiece from the 2D image data provided by the vision system. Conventional methods face the complexity of model construction and time consumption on geometric matching. To overcome these difficulties, this paper proposes a search-based method to determine appropriate model and pose of a 3D workpiece that match the 2D image data. Concretely, we formulate the above problem as an optimization problem aiming at finding appropriate model parameters and pose parameters which minimizes the error between the notional 2D image (given by the model/pose parameters being optimized) and the real 2D image (provided by the vision system). Due to the coupling of model and pose parameters and discontinuity of the objective function, the above optimization problem cannot be tackled by conventional optimization techniques. Hence, we employ an evolutionary algorithm to cope with the optimization problem, where the evolutionary algorithm utilizes our problem-specific knowledge and adopts a hierarchical coarse-to-fine style to meet the requirement of online estimation. Experimental results demonstrate that our method is effective and efficient.