Introducing a new problem: shape-from-silhouette when the relative positions of the viewpoints is unknown

3D shapes can be reconstructed from 2D silhouettes by back-projecting them from the corresponding viewpoints and intersecting the resulting solid cones. However, in many practical cases as observing an aircraft or an asteroid, the positions of the viewpoints with respect to the object are not known. In these cases, the relative position of the solid cones is not known and the intersection cannot be performed. The purpose of this paper is introducing and stating in a theoretical framework the problem of understanding 3D shapes from silhouettes when the relative positions of the viewpoints are unknown. The results presented provide a first insight into the problem. In particular, the case of orthographic viewing directions parallel to the same plane is thoroughly discussed, and sets of inequalities are presented which allow determining objects compatible with the silhouettes.     

FORMS: A flexible object recognition and modelling system

We describe a flexible object recognition and modelling system (FORMS) which represents and recognizes animate objects from their silhouettes. This consists of a model for generating the shapes of animate objects which gives a formalism for solving the inverse problem of object recognition. We model all objects at three levels of complexity: (i) the primitives, (ii) the mid-grained shapes, which are deformations of the primitives, and (iii) objects constructed by using a grammar to join mid-grained shapes together. The deformations of the primitives can be characterized by principal component analysis or modal analysis. When doing recognition the representations of these objects are obtained in a bottom-up manner from their silhouettes by a novel method for skeleton extraction and part segmentation based on deformable circles. These representations are then matched to a database of prototypical objects to obtain a set of candidate interpretations. These interpretations are verified in a top-down process. The system is demonstrated to be stable in the presence of noise, the absence of parts, the presence of additional parts, and considerable variations in articulation and viewpoint. Finally, we describe how such a representation scheme can be automatically learnt from examples.     

A three-dimensional criterion for the determination of optimal grip points

Multifingered grasping has aroused remarkable interest because it makes possible the manipulation of objects of different shapes and sizes. However, manipulating and picking up objects in unstructured environments requires accurate contact-point selection. Generally, such processes are subject to external forces which are difficult to predict and may change during task execution.In this paper, an optimization criterion is proposed which is meant to select the optimal grip points in a three-dimensional problem for any number of contact points. This method may be applied to three-dimensional objects of any shape (curved or polygonal) and does not require that the external forces acting on the object be known. A grasp quality index is presented which has been obtained by minimizing the grasping forces required to balance a generalized external disturbance. The optimization criterion has led to the formulation of a single optimization problem with non-linear constraints. Finally, the paper presents the results obtained in searches for the optimal grip points on some two- and three-dimensional objects.     

Multiview photometric stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.     

Real‐time Rendering of Heterogeneous Translucent Objects with Arbitrary Shapes

We present a real‐time algorithm for rendering translucent objects of arbitrary shapes. We approximate the scattering of light inside the objects using the diffusion equation, which we solve on‐the‐fly using the GPU. Our algorithm is general enough to handle arbitrary geometry, heterogeneous materials, deformable objects and modifications of lighting, all in real‐time. In a pre‐processing step, we discretize the object into a regular 4‐connected structure (QuadGraph). Due to its regular connectivity, this structure is easily packed into a texture and stored on the GPU. At runtime, we use the QuadGraph stored on the GPU to solve the diffusion equation, in real‐time, taking into account the varying input conditions: Incoming light, object material and geometry. We handle deformable objects, provided the deformation does not change the topological structure of the objects.     

Curve and Surface Duals and the Recognition of Curved 3D Objects from their Silhouettes

This article addresses the problem of recognizing a solid bounded by a smooth surface in a single image. The proposed approach is based on a new representation for two- and three-dimensional shapes, called their signature, that exploits the close relationship between the dual of a surface and the dual of its silhouette in weak-perspective images. Objects are modeled by rotating them in front of a camera without any knowledge of or constraints on their motion. The signatures of their silhouettes are concatenated into a single object signature. To recognize an object from novel viewpoint other than those used during modeling, the signature of the contours extracted from a test photograph is matched to the signatures of all modeled objects signatures. This approach has been implemented, and recognition examples are presented.     

Matching three-dimensional objects using silhouettes

A method for matching three-dimensional objects against a library of models from an observed sequence of silhouettes is presented in this correspondence. Based upon the observed silhouettes, the three-dimensional structure of the object is constructed and refined. The principal moments and three primary silhouettes are computed for the constructed three-dimensional objects to represent the aggregate and detailed structure parameters. The adaptive matching technique requires that sufficient silhouettes be added to modify the structure of the unknown object until consistent and steady matching results are obtained. The library for matching is based on three primary silhouettes of the model objects. Experiments conducted show a fast convergence to a consistent result may be achieved provided that a reasonable choice of silhouettes is made.     

Learning to Transfer In-Hand Manipulations Using a Greedy Shape Curriculum

In-hand object manipulation is challenging to simulate due to complex contact dynamics, non-repetitive finger gaits, and the need to indirectly control unactuated objects. Further adapting a successful manipulation skill to new objects with different shapes and physical properties is a similarly challenging problem. In this work, we show that natural and robust in-hand manipulation of simple objects in a dynamic simulation can be learned from a high quality motion capture example via deep reinforcement learning with careful designs of the imitation learning problem. We apply our approach on both single-handed and two-handed dexterous manipulations of diverse object shapes and motions. We then demonstrate further adaptation of the example motion to a more complex shape through curriculum learning on intermediate shapes morphed between the source and target object. While a naive curriculum of progressive morphs often falls short, we propose a simple greedy curriculum search algorithm that can successfully apply to a range of objects such as a teapot, bunny, bottle, train, and elephant.     

Visual Recognition and Categorization on the Basis of Similarities to Multiple Class Prototypes

One of the difficulties of object recognition stems from the need to overcome the variability in object appearance caused by pose and other factors, such as illumination. The influence of these factors can be countered by learning to interpolate between stored views of the target object, taken under representative combinations of viewing conditions. Difficulties of another kind arise in daily life situations that require categorization, rather than recognition, of objects. Although categorization cannot rely on interpolation between stored examples, we show that knowledge of several representative members, or prototypes, of each of the categories of interest can provide the necessary computational substrate for the categorization of new instances. We describe a system that represents input shapes by their similarities to several prototypical objects, and show that it can recognize new views of the familiar objects, discriminate among views of previously unseen shapes, and attribute the latter to familiar categories.     

Robust recovery of shapes with unknown topology from the dual space

In this paper, we address the problem of reconstructing an object surface from silhouettes. Previous works by other authors have shown that, based on the principle of duality, surface points can be recovered, theoretically, as the dual to the tangent plane space of the object. In practice, however, the identification of tangent basis in the tangent plane space is not trivial given a set of discretely sampled data. This problem is further complicated by the existence of bi-tangents to the object surface. The key contribution of this paper is the introduction of epipolar parameterization in identifying a well-defined local tangent basis. This extends the applicability of existing dual space reconstruction methods to fairly complicated shapes, without making any explicit assumption on the object topology. We verify our approach with both synthetic and real-world data, and compare it both qualitatively and quantitatively with other popular reconstruction algorithms. Experimental results demonstrate that our proposed approach produces more accurate estimation, whilst maintaining reasonable robustness towards shapes with complex topologies.     

How far 3D shapes can be understood from 2D silhouettes

Each 2D silhouette of a 3D unknown object O constrains O inside the volume obtained by back-projecting the silhouette from the corresponding viewpoint. A set of silhouettes specifies a boundary volume R obtained by intersecting the volumes due to each silhouette. R more or less closely approximates O, depending on the viewpoints and the object itself. This approach to the reconstruction of 3D objects is usually referred to as volume intersection. This correspondence addresses the problem of inferring the shape of the unknown object O from the reconstructed object R. For doing this, the author divides the points of the surface of R into hard points, which belong to the surface of any possible object originating R, and soft points, which may or may not belong to O. The author considers two cases: In the first case R is the closest approximation of O which can be obtained from its silhouettes, i.e., its visual hull; in the second case, R is a generic reconstructed object. In both cases the author supplies necessary and sufficient conditions for a point to be hard and gives rules for computing the hard surfaces     

Dynamical statistical shape priors for level set-based tracking

In recent years, researchers have proposed introducing statistical shape knowledge into level set-based segmentation methods in order to cope with insufficient low-level information. While these priors were shown to drastically improve the segmentation of familiar objects, so far the focus has been on statistical shape priors which are static in time. Yet, in the context of tracking deformable objects, it is clear that certain silhouettes (such as those of a walking person) may become more or less likely over time. In this paper, we tackle the challenge of learning dynamical statistical models for implicitly represented shapes. We show how these can be integrated as dynamical shape priors in a Bayesian framework for level set-based image sequence segmentation. We assess the effect of such shape priors "with memory" on the tracking of familiar deformable objects in the presence of noise and occlusion. We show comparisons between dynamical and static shape priors, between models of pure deformation and joint models of deformation and transformation, and we quantitatively evaluate the segmentation accuracy as a function of the noise level and of the camera frame rate. Our experiments demonstrate that level set-based segmentation and tracking can be strongly improved by exploiting the temporal correlations among consecutive silhouettes which characterize deforming shapes.     

Actions as space-time shapes

Human action in video sequences can be seen as silhouettes of a moving torso and protruding limbs undergoing articulated motion. We regard human actions as three-dimensional shapes induced by the silhouettes in the space-time volume. We adopt a recent approach for analyzing 2D shapes and generalize it to deal with volumetric space-time action shapes. Our method utilizes properties of the solution to the Poisson equation to extract space-time features such as local space-time saliency, action dynamics, shape structure and orientation. We show that these features are useful for action recognition, detection and clustering. The method is fast, does not require video alignment and is applicable in (but not limited to) many scenarios where the background is known. Moreover, we demonstrate the robustness of our method to partial occlusions, non-rigid deformations, significant changes in scale and viewpoint, high irregularities in the performance of an action, and low quality video.     

A Multibody Factorization Method for Independently Moving Objects

The structure-from-motion problem has been extensively studied in the field of computer vision. Yet, the bulk of the existing work assumes that the scene contains only a single moving object. The more realistic case where an unknown number of objects move in the scene has received little attention, especially for its theoretical treatment. In this paper we present a new method for separating and recovering the motion and shape of multiple independently moving objects in a sequence of images. The method does not require prior knowledge of the number of objects, nor is dependent on any grouping of features into an object at the image level. For this purpose, we introduce a mathematical construct of object shapes, called the shape interaction matrix, which is invariant to both the object motions and the selection of coordinate systems. This invariant structure is computable solely from the observed trajectories of image features without grouping them into individual objects. Once the matrix is computed, it allows for segmenting features into objects by the process of transforming it into a canonical form, as well as recovering the shape and motion of each object. The theory works under a broad set of projection models (scaled orthography, paraperspective and affine) but they must be linear, so it excludes projective cameras.     

Shape Morphing-Based Control of Robotic Visual Servoing

We present an approach for controlling robotic interactions with objects, using synthetic images generated by morphing shapes. In particular, we attempt the problem of positioning an eye-in-hand robotic system with respect to objects in the workspace for grasping and manipulation. In our formulation, the grasp position (and consequently the approach trajectory of the manipulator), varies with each object. The proposed solution to the problem consists of two parts. First, based on a model-based object recognition framework, images of the objects taken at the desired grasp pose are stored in a database. The recognition and identification of the grasp position for an unknown input object (selected from the family of recognizable objects) occurs by morphing its contour to the templates in the database and using the virtual energy spent during the morph as a dissimilarity measure. In the second step, the images synthesized during the morph are used to guide the eye-in-hand system and execute the grasp. The proposed method requires minimal calibration of the system. Furthermore, it conjoins techniques from shape recognition, computer graphics, and vision-based robot control in a unified engineering amework. Potential applications range from recognition and positioning with respect to partially-occluded or deformable objects to planning robotic grasping based on human demonstration.     

Moving Objects Localization by Local Regions Based Level Set: Application on Urban Traffic

In this paper, a novel method for locating multiple moving objects in a video sequences captured by a stationary camera is proposed. In order to determine the precise location of the objects in an image, a new local regions based level set model is carried out. The whole process consists of two main parts: the global detection and the fine localization. During the global detection, the presence or absence of an object in an image is determined by the Mixture of Gaussians method. For the fine localization, we propose to reformulate global energies by utilizing little squared windows centered on each point over a thin band surrounding the zero level set, hence the object contour can be reshaped into small local interior and exterior regions that lead to compute a family of adaptive local energies, which enables us to well localize the moving objects. Moreover, we propose to adapt the smoothness of the contours, and the accuracy of the objects’ perimeter of different shapes with an automatic stopping criterion. The proposed method has been tested on different real urban traffic videos, and the experiment results demonstrate that our algorithm can locate effectively and accurately the moving objects; optimize the results of the localized objects and also decrease the computations load.     

Proximity Queries Between Convex Objects: An Interior Point Approach for Implicit Surfaces

This paper presents a general method for exact distance computation between convex objects represented as intersections of implicit surfaces. Exact distance computation algorithms are particularly important for applications involving objects that make intermittent contact, such as in dynamic simulations and in haptic interactions. They can also be used in the narrow phase of hierarchical collision detection. In contrast to geometric approaches developed for polyhedral objects, we formulate the distance computation problem as a convex optimization problem. We use an interior point method to solve the optimization problem and demonstrate that, for general convex objects represented as implicit surfaces, interior point approaches are globally convergent, and fast in practice. Further, they provide polynomial-time guarantees for implicit surface objects when the implicit surfaces have self-concordant barrier functions. We use a primal-dual interior point algorithm that solves the Karush-Kuhn-Tucker (KKT) conditions obtained from the convex programming formulation. For the case of polyhedra and quadrics, we establish a theoretical time complexity of O(n^1.5), where n is the number of constraints. We present implementation results for example implicit surface objects, including polyhedra, quadrics, and generalizations of quadrics such as superquadrics and hyperquadrics, as well as intersections of these surfaces. We demonstrate that in practice, the algorithm takes time linear in the number of constraints, and that distance computation rates of about 1 kHz can be achieved. We also extend the approach to proximity queries between deforming convex objects. Finally, we show that continuous collision detection for linearly translating objects can be performed by solving two related convex optimization problems. For polyhedra and quadrics, we establish that the computational complexity of this problem is also O(n^1.5).     

Shape-based indexing scheme for camera view invariant 3-D object retrieval

Camera view invariant 3-D object retrieval is an important issue in many traditional and emerging applications such as security, surveillance, computer-aided design (CAD), virtual reality, and place recognition. One straightforward method for camera view invariant 3-D object retrieval is to consider all the possible camera views of 3-D objects. However, capturing and maintaining such views require an enormous amount of time and labor. In addition, all camera views should be indexed for reasonable retrieval performance, which requires extra storage space and maintenance overhead. In the case of shape-based 3-D object retrieval, such overhead could be relieved by considering the symmetric shape feature of most objects. In this paper, we propose a new shape-based indexing and matching scheme of real or rendered 3-D objects for camera view invariant object retrieval. In particular, in order to remove redundant camera views to be indexed, we propose a camera view skimming scheme, which includes: i) mirror shape pairing and ii) camera view pruning according to the symmetrical patterns of object shapes. Since our camera view skimming scheme considerably reduces the number of camera views to be indexed, it could relieve the storage requirement and improve the matching speed without sacrificing retrieval accuracy. Through various experiments, we show that our proposed scheme can achieve excellent performance.