Transparent surface modeling from a pair of polarization images

We propose a method for measuring surface shapes of transparent objects by using a polarizing filter. Generally, the light reflected from an object is partially polarized. The degree of polarization depends upon the incident angle, which, in turn, depends upon the surface normal. Therefore, we can obtain surface normals of objects by observing the degree of polarization at each surface point. Unfortunately, the correspondence between the degree of polarization and the surface normal is not one to one. Hence, to obtain the correct surface normal, we have to solve the ambiguity problem. In this paper, we introduce a method to solve the ambiguity by comparing the polarization data in two objects, i.e., normal position and tilted with small angle position. We also discuss the geometrical features of the object surface and propose a method for matching two sets of polarization data at identical points on the object surface.     

Dense Reconstruction of Transparent Objects by Altering Incident Light Paths Through Refraction

This paper addresses the problem of reconstructing the surface shape of transparent objects. The difficulty of this problem originates from the viewpoint dependent appearance of a transparent object, which quickly makes reconstruction methods tailored for diffuse surfaces fail disgracefully. In this paper, we introduce a fixed viewpoint approach to dense surface reconstruction of transparent objects based on refraction of light. We present a simple setup that allows us to alter the incident light paths before light rays enter the object by immersing the object partially in a liquid, and develop a method for recovering the object surface through reconstructing and triangulating such incident light paths. Our proposed approach does not need to model the complex interactions of light as it travels through the object, neither does it assume any parametric form for the object shape nor the exact number of refractions and reflections taken place along the light paths. It can therefore handle transparent objects with a relatively complex shape and structure, with unknown and inhomogeneous refractive index. We also show that for thin transparent objects, our proposed acquisition setup can be further simplified by adopting a single refraction approximation. Experimental results on both synthetic and real data demonstrate the feasibility and accuracy of our proposed approach.     

Novel View Synthesis Of Transparent Object From a Single Image

We propose a method for converting a single image of a transparent object into multi-view photo that enables users observing the object from multiple new angles, without inputting any 3D shape. The complex light paths formed by refraction and reflection makes it challenging to compute the lighting effects of transparent objects from a new angle. We construct an encoder–decoder network for normal reconstruction and texture extraction, which enables synthesizing novel views of transparent object from a set of new views and new environment maps using only one RGB image. By simultaneously considering the optical transmission and perspective variation, our network learns the characteristics of optical transmission and the change of perspective as guidance to the conversion from RGB colours to surface normals. A texture extraction subnetwork is proposed to alleviate the contour loss phenomenon during normal map generation. We test our method using 3D objects within and without our training data, including real 3D objects that exists in our lab, and completely new environment maps that we take using our phones. The results show that our method performs better on view synthesis of transparent objects in complex scenes using only a single-view image.     

Shape Based Detection and Top-Down Delineation Using Image Segments

We introduce a segmentation-based detection and top-down figure-ground delineation algorithm. Unlike common methods which use appearance for detection, our method relies primarily on the shape of objects as is reflected by their bottom-up segmentation. Our algorithm receives as input an image, along with its bottom-up hierarchical segmentation. The shape of each segment is then described both by its significant boundary sections and by regional, dense orientation information derived from the segment’s shape using the Poisson equation. Our method then examines multiple, overlapping segmentation hypotheses, using their shape and color, in an attempt to find a “coherent whole,” i.e., a collection of segments that consistently vote for an object at a single location in the image. Once an object is detected, we propose a novel pixel-level top-down figure-ground segmentation by “competitive coverage” process to accurately delineate the boundaries of the object. In this process, given a particular detection hypothesis, we let the voting segments compete for interpreting (covering) each of the semantic parts of an object. Incorporating competition in the process allows us to resolve ambiguities that arise when two different regions are matched to the same object part and to discard nearby false regions that participated in the voting process. We provide quantitative and qualitative experimental results on challenging datasets. These experiments demonstrate that our method can accurately detect and segment objects with complex shapes, obtaining results comparable to those of existing state of the art methods. Moreover, our method allows us to simultaneously detect multiple instances of class objects in images and to cope with challenging types of occlusions such as occlusions by a bar of varying size or by another object of the same class, that are difficult to handle with other existing class-specific top-down segmentation methods.     

Photometric Stereo with General,Unknown Lighting

Work on photometric stereo has shown how to recover the shape and reflectance properties of an object using multiple images taken with a fixed viewpoint and variable lighting conditions. This work has primarily relied on known lighting conditions or the presence of a single point source of light in each image. In this paper we show how to perform photometric stereo assuming that all lights in a scene are distant from the object but otherwise unconstrained. Lighting in each image may be an unknown and may include arbitrary combination of diffuse, point and extended sources. Our work is based on recent results showing that for Lambertian objects, general lighting conditions can be represented using low order spherical harmonics. Using this representation we can recover shape by performing a simple optimization in a low-dimensional space. We also analyze the shape ambiguities that arise in such a representation. We demonstrate our method by reconstructing the shape of objects from images obtained under a variety of lightings. We further compare the reconstructed shapes against shapes obtained with a laser scanner.     

Modeling of shape bifurcation phenomena in manipulations of deformable string objects

A systematic approach to the modeling of deformable string objects is presented. Various string objects such as cords and wires are manipulated in many manufacturing processes. In such processes, it is important for successful manipulation to evaluate their shapes on a computer in advance because their shapes can be changed easily even under the same conditions. We refer to the situation that a shape can be changed into another shape even under the same constraints as shape bifurcation. In this paper, we will develop an analytical model of the shape of string objects including shape bifurcation. First, we will investigate the mechanism of the shape bifurcation phenomena based on the potential energy. Then, we will propose a hypothesis on the mechanism of shape bifurcation based on local minima of the potential energy. Second, the potential energy of a string object and the geometric constraints imposed on it are formulated. The shape of the object can be derived by minimizing the potential energy under the geometric constraints. Thirdly, a procedure to compute the shape of a deformed string object is developed considering the local minima of the energy. Finally, we show some numerical examples with shape bifurcation using our proposed method. From the results, we conclude that our proposed method accurately describes deformed shapes of string objects including shape bifurcation.     

Visual saliency guided textured model simplification

Mesh geometry can be used to model both object shape and details. If texture maps are involved, it is common to let mesh geometry mainly model object shapes and let the texture maps model the most object details, optimising data size and complexity of an object. To support efficient object rendering and transmission, model simplification can be applied to reduce the modelling data. However, existing methods do not well consider how object features are jointly represented by mesh geometry and texture maps, having problems in identifying and preserving important features for simplified objects. To address this, we propose a visual saliency detection method for simplifying textured 3D models. We produce good simplification results by jointly processing mesh geometry and texture map to produce a unified saliency map for identifying visually important object features. Results show that our method offers a better object rendering quality than existing methods.     

Object tracking method based on data computing

The data computing process is utilized in various areas such as autonomous driving. Autonomous vehicles are intended to detect and track nearby moving objects avoiding collisions and to navigate in complex situations, such as heavy traffic and dense pedestrian areas. Therefore, object tracking is the core technology in the environment perception systems of autonomous vehicles and requires the monitoring of surrounding objects and the prediction of the moving states of objects in real time. In this paper, a multiple object tracking method based on light detection and ranging (LiDAR) data is proposed by using a Kalman filter and data computing process. We suppose that the movements of the tracking objects are captured consecutively as frames; thus, model-based detection and tracking of dynamic objects are possible. A Kalman filter is applied for predicting posterior state of tracking object based on anterior state of the tracking object. State denotes the positions, shapes, and sizes of objects. By computing the likelihood probability between predicted tracking objects and clusters which registered from tracking objects, the data association process of the tracking objects can be generated. Experimental results showed enhanced object tracking performance in a dynamic environment. The average matching probability of the tracking object was greater than 92.9%.

     

Shape understanding system: Understanding the thin object

This paper presents the method of understanding objects that can be considered as thin objects. The proposed method of understanding thin objects is part of the shape understanding method developed by the author. The main novelty of the presented method is that the process of understanding thin objects is related to the visual concept represented as a symbolic name of the possible class of shapes. The possible classes of shape, viewed as hierarchical structures, are incorporated into the shape model. At each stage of the reasoning process that led to assigning of an examined object to one of the possible classes, novel processing methods are used. These methods are very efficient because they deal with a very specific class of shapes. In this paper, the 2-D objects that are classified as thin objects are regarded as geometrical objects without any reference to the real world objects. However, the shape under standing method is designed to understand an object at many levels of interpretation, such as the topological level, the linguistic level and the real world reference level. This approach influences the way in which the system of shape understanding is designed. The system consists of different types of experts that perform different processing and reasoning tasks.     

An Investigation into the Use of Physical Modeling for the Prediction of Various Feature Types Visible from Different Viewpoints

Given that aspect graph and viewsphere-based object recognition systems provide a valid mechanism for 3D object recognition of man-made objects, this paper provides a flexible, automated, and general purpose technique for generating the view information for each viewpoint. An advantage of the work is that the technique is unaffected by object complexity because each step makes no assumptions about object shape. The only limitation is that the object can be described by a boundary representation. A second advantage is that the technique can include other feature types such as specularity. The reason for this is that raytracing techniques are used to simulate the physical process of image generation. Hence it is extendible to visible features resulting from effects due to lighting, surface texture, color, transparency, etc. The work described in this paper shows how occluding and nonoccluding edge-based features can be extracted using image processing techniques and then parametrized and also how regions of specularity can be predicted and described. The use of physical modeling enables situations to be simulated and predicted that are intractable for CAD-based methods (e.g., multiscale feature prediction). An advantage of the method is that the interface between the technique and the raytracing module is a rendered image. Should better physics-based image formation algorithms become available, then they could replace the raytracing module with little modification to the rest of the method.     

Recognizing objects with multiple configurations

Computer vision has been extensively adopted in many domains during the last three decades. One of the main goals of computer vision applications is to recognize objects. Generally, computers can successfully achieve object recognition by relying on a large quantity of data. In real world, some objects may own diverse configurations or/and be observed at various angles and positions, and the process of object recognition is denoted as recognizing objects in dynamic state. It is difficult to collect enough data to achieve the sorts of objects recognition. In order to resolve the problem, we propose a technique to achieve object recognition which is not only in static state where the objects do not own multiple configurations, but also in dynamic state. First, we apply an effective robust algorithm to obtain landmarks from objects in two dimensional images. With the algorithm, the number of landmarks from different objects can be appointed in advance. A set of landmarks as a point is projected into a pre-shape space and a shape space. Next, a method is proposed to create a surface among three basic data models in a pre-shape space. If basic data are too few to create a surface or a curve, a new basic data can be built from the basic data. Then, a series of new data models can be obtained from these basic data in a pre-shape space. Finally, object recognition can be achieved by using the new data models in shape space. We give some examples to show the algorithms are efficient not only for the objects with noises, but also for the ones with various configurations.     

Regressing Local to Global Shape Properties for Online Segmentation and Tracking

We propose a novel regression based framework that uses online learned shape information to reconstruct occluded object contours. Our key insight is to regress the global, coarse, properties of shape from its local properties, i.e. its details. We do this by representing shapes using their 2D discrete cosine transforms and by regressing low frequency from high frequency harmonics. We learn this regression model using Locally Weighted Projection Regression which expedites online, incremental learning. After sufficient observation of a set of unoccluded shapes, the learned model can detect occlusion and recover the full shapes from the occluded ones. We demonstrate the ideas using a level-set based tracking system that provides shape and pose, however, the framework could be embedded in any segmentation-based tracking system. Our experiments demonstrate the efficacy of the method on a variety of objects using both real data and artificial data.     

Transparent and Specular Object Reconstruction

This state of the art report covers reconstruction methods for transparent and specular objects or phenomena. While the 3D acquisition of opaque surfaces with Lambertian reflectance is a well‐studied problem, transparent, refractive, specular and potentially dynamic scenes pose challenging problems for acquisition systems. This report reviews and categorizes the literature in this field. Despite tremendous interest in object digitization, the acquisition of digital models of transparent or specular objects is far from being a solved problem. On the other hand, real‐world data is in high demand for applications such as object modelling, preservation of historic artefacts and as input to data‐driven modelling techniques. With this report we aim at providing a reference for and an introduction to the field of transparent and specular object reconstruction. We describe acquisition approaches for different classes of objects. Transparent objects/phenomena that do not change the straight ray geometry can be found foremost in natural phenomena. Refraction effects are usually small and can be considered negligible for these objects. Phenomena as diverse as fire, smoke, and interstellar nebulae can be modelled using a straight ray model of image formation. Refractive and specular surfaces on the other hand change the straight rays into usually piecewise linear ray paths, adding additional complexity to the reconstruction problem. Translucent objects exhibit significant sub‐surface scattering effects rendering traditional acquisition approaches unstable. Different classes of techniques have been developed to deal with these problems and good reconstruction results can be achieved with current state‐of‐the‐art techniques. However, the approaches are still specialized and targeted at very specific object classes. We classify the existing literature and hope to provide an entry point to this exiting field.     

Multimorphing: A tool for shape synthesis and analysis

Morphing is a shape transformation where the shape of one object is deformed to the shape of the other object. It is used as an animation or a modeling technique. Classical morphing operates between two input objects but this concept can be extended to multiple input objects – the so called multimorphing. Shapes generated by the multimorphing form a space of shapes motivated by an affine space. Besides the analogy with an affine space we also introduce an inner product and a concept of an orthogonal projection. We also show how to explore space of shapes and how to systematically generate new shapes. The paper focuses on the boundary representation, although some ideas are more general and can be used for other representations, too.     

Nonrigid structure-from-motion: estimating shape and motion with hierarchical Priors

This paper describes methods for recovering time-varying shape and motion of non-rigid 3D objects from uncalibrated 2D point tracks. For example, given a video recording of a talking person, we would like to estimate the 3D shape of the face at each instant, and learn a model of facial deformation. Time-varying shape is modeled as a rigid transformation combined with a non-rigid deformation. Reconstruction is ill-posed if arbitrary deformations are allowed, and thus additional assumptions about deformations are required. We first suggest restricting shapes to lie within a low-dimensional subspace, and describe estimation algorithms. However, this restriction alone is insufficient to constrain reconstruction. To address these problems, we propose a reconstruction method using a Probabilistic Principal Components Analysis (PPCA) shape model, and an estimation algorithm that simultaneously estimates 3D shape and motion for each instant, learns the PPCA model parameters, and robustly fills-in missing data points. We then extend the model to model temporal dynamics in object shape, allowing the algorithm to robustly handle severe cases of missing data.     

Acquiring a complete 3D model from specular motion under theillumination of circular-shaped light sources

We recover 3D models of objects with specular surfaces. An object is rotated and its continuous images are taken. Circular-shaped light sources that generate conic rays are used to illuminate the rotating object in such a way that highlighted stripes can be observed on most of the specular surfaces. Surface shapes can be computed from the motions of highlights in the continuous images; either specular motion stereo or single specular trace mode can be used. When the lights are properly set, each point on the object can be highlighted during the rotation. The shape for each rotation plane is measured independently using its corresponding epipolar plane image. A 3D shape model is subsequently reconstructed by combining shapes at different rotation planes. Computing a shape is simple and requires only the motion of highlight on each rotation plane. The novelty of this paper is the complete modeling of a general type of specular objects that has not been accomplished before     

Recognition of multiple configurations of objects with limited data

One of the main goals of image understanding and computer vision applications is to recognize an object from various images. A lot of studies on recognizing objects based on invariable shapes have been explored, however, in reality, there are many objects with multiple configurations, which are very difficult to be recognized. We call this kind of problem as the recognition of multiple configurations of objects (RMCO). To achieve RMCO, firstly we obtain a shortest path (the Geodesic distance path) between two feature vectors in pre-shape spaces; along this obtained path, we can generate a series of data which can be used to recognize the observed objects by using shape space theories. In other words, we may augment the database content with very limited data to recognize more objects.     

Instance-based object recognition in 3D point clouds using discriminative shape primitives

3D local shapes are a critical cue for object recognition in 3D point clouds. This paper presents an instance-based 3D object recognition method via informative and discriminative shape primitives. We propose a shape primitive model that measures geometrical informativity and discriminativity of 3D local shapes of an object. Discriminative shape primitives of the object are extracted automatically by model parameter optimization. We achieve object recognition from 2.5/3D scenes via shape primitive classification and recover the 3D poses of the identified objects simultaneously. The effectiveness and the robustness of the proposed method were verified on popular instance-based 3D object recognition datasets. The experimental results show that the proposed method outperforms some existing instance-based 3D object recognition pipelines in the presence of noise, varying resolutions, clutter and occlusion.     

Object matching using deformable templates

We propose a general object localization and retrieval scheme based on object shape using deformable templates. Prior knowledge of an object shape is described by a prototype template which consists of the representative contour/edges, and a set of probabilistic deformation transformations on the template. A Bayesian scheme, which is based on this prior knowledge and the edge information in the input image, is employed to find a match between the deformed template and objects in the image. Computational efficiency is achieved via a coarse-to-fine implementation of the matching algorithm. Our method has been applied to retrieve objects with a variety of shapes from images with complex background. The proposed scheme is invariant to location, rotation, and moderate scale changes of the template     

Image Warping for Shape Recovery and Recognition

We demonstrate that, for a large class of reflectance functions, there is a direct relationship between image warps and the corresponding geometric deformations of the underlying three-dimensional objects. This helps explain the hidden geometrical assumptions in object recognition schemes which involve two-dimensional image warping computed by matching image intensity. In addition, it allows us to propose a novel variant of shape from shading which we call shape from image warping. The idea is that the three-dimensional shape of an object is estimated by determining how much the image of the object is warped with respect to the image of a known prototype shape. Therefore, detecting the image warp relative to a prototype of known shape allows us to reconstruct the shape of the imaged object. We derive properties of these shape warps and illustrate the results by recovering the shapes of faces.