A Neural Network Approach to Planar-Object Recognition in 3D Space

Most existing 2D object recognition algorithms are not perspective (or projective) invariant, and hence are not suitable for many real-world applications. By contrast, one of the primary goals of this research is to develop a flat object matching system that can identify and localise an object, even when seen from different viewpoints in 3D space. In addition, we also strive to achieve good scale invariance and robustness against partial occlusion as in any practical 2D object recognition system. The proposed system uses multi-view model representations and objects are recognised by self-organised dynamic link matching. The merit of this approach is that it offers a compact framework for concurrent assessments of multiple match hypotheses by promoting competitions and/or co-operations among several local mappings of model and test image feature correspondences. Our experiments show that the system is very successful in recognising object to perspective distortion, even in rather cluttered scenes. Receiveed: 29 May 1998?,Received in revised form: 12 October 1998?Accepted: 26 October 1998     

Silhouette-based occluded object recognition through curvature scale space

A complete and practical system for occluded object recognition has been developed which is very robust with respect to noise and local deformations of shape (due to weak perspective distortion, segmentation errors and non-rigid material) as well as scale, position and orientation changes of the objects. The system has been tested on a wide variety of free-form 3D objects. An industrial application is envisaged where a fixed camera and a light-box are utilized to obtain images. Within the constraints of the system, every rigid 3D object can be modeled by a limited number of classes of 2D contours corresponding to the object's resting positions on the light-box. The contours in each class are related to each other by a 2D similarity transformation. The Curvature Scale Space technique [26, 28] is then used to obtain a novel multi-scale segmentation of the image and the model contours. Object indexing [16, 32, 36] is used to narrow down the search space. An efficient local matching algorithm is utilized to select the best matching models. Received: 5 August 1996 / Accepted: 19 March 1997     

Discriminative Sketch‐based 3D Model Retrieval via Robust Shape Matching

We propose a sketch‐based 3D shape retrieval system that is substantially more discriminative and robust than existing systems, especially for complex models. The power of our system comes from a combination of a contour‐based 2D shape representation and a robust sampling‐based shape matching scheme. They are defined over discriminative local features and applicable for partial sketches; robust to noise and distortions in hand drawings; and consistent when strokes are added progressively. Our robust shape matching, however, requires dense sampling and registration and incurs a high computational cost. We thus devise critical acceleration methods to achieve interactive performance: precomputing kNN graphs that record transformations between neighboring contour images and enable fast online shape alignment; pruning sampling and shape registration strategically and hierarchically; and parallelizing shape matching on multi‐core platforms or GPUs. We demonstrate the effectiveness of our system through various experiments, comparisons, and user studies.     

Weakly Supervised Part-wise 3D Shape Reconstruction from Single-View RGB Images

In order for the deep learning models to truly understand the 2D images for 3D geometry recovery, we argue that single-view reconstruction should be learned in a part-aware and weakly supervised manner. Such models lead to more profound interpretation of 2D images in which part-based parsing and assembling are involved. To this end, we learn a deep neural network which takes a single-view RGB image as input, and outputs a 3D shape in parts represented by 3D point clouds with an array of 3D part generators. In particular, we devise two levels of generative adversarial network (GAN) to generate shapes with both correct part shape and reasonable overall structure. To enable a self-taught network training, we devise a differentiable projection module along with a self-projection loss measuring the error between the shape projection and the input image. The training data in our method is unpaired between the 2D images and the 3D shapes with part decomposition. Through qualitative and quantitative evaluations on public datasets, we show that our method achieves good performance in part-wise single-view reconstruction.     

CustomCut: On‐demand Extraction of Customized 3D Parts with 2D Sketches

Several applications in shape modeling and exploration require identification and extraction of a 3D shape part matching a 2D sketch. We present CustomCut, an on‐demand part extraction algorithm. Given a sketched query, CustomCut automatically retrieves partially matching shapes from a database, identifies the region optimally matching the query in each shape, and extracts this region to produce a customized part that can be used in various modeling applications. In contrast to earlier work on sketch‐based retrieval of predefined parts, our approach can extract arbitrary parts from input shapes and does not rely on a prior segmentation into semantic components. The method is based on a novel data structure for fast retrieval of partial matches: the randomized compound k‐NN graph built on multi‐view shape projections. We also employ a coarse‐to‐fine strategy to progressively refine part boundaries down to the level of individual faces. Experimental results indicate that our approach provides an intuitive and easy means to extract customized parts from a shape database, and significantly expands the design space for the user. We demonstrate several applications of our method to shape design and exploration.     

Learning Generative Models of 3D Structures

3D models of objects and scenes are critical to many academic disciplines and industrial applications. Of particular interest is the emerging opportunity for 3D graphics to serve artificial intelligence: computer vision systems can benefit from synthetically-generated training data rendered from virtual 3D scenes, and robots can be trained to navigate in and interact with real-world environments by first acquiring skills in simulated ones. One of the most promising ways to achieve this is by learning and applying generative models of 3D content: computer programs that can synthesize new 3D shapes and scenes. To allow users to edit and manipulate the synthesized 3D content to achieve their goals, the generative model should also be structure-aware: it should express 3D shapes and scenes using abstractions that allow manipulation of their high-level structure. This state-of-the-art report surveys historical work and recent progress on learning structure-aware generative models of 3D shapes and scenes. We present fundamental representations of 3D shape and scene geometry and structures, describe prominent methodologies including probabilistic models, deep generative models, program synthesis, and neural networks for structured data, and cover many recent methods for structure-aware synthesis of 3D shapes and indoor scenes.     

A Theory of Shape by Space Carving

In this paper we consider the problem of computing the 3D shape of an unknown, arbitrarily-shaped scene from multiple photographs taken at known but arbitrarily-distributed viewpoints. By studying the equivalence class of all 3D shapes that reproduce the input photographs, we prove the existence of a special member of this class, the photo hull, that (1) can be computed directly from photographs of the scene, and (2) subsumes all other members of this class. We then give a provably-correct algorithm, called Space Carving, for computing this shape and present experimental results on complex real-world scenes. The approach is designed to (1) capture photorealistic shapes that accurately model scene appearance from a wide range of viewpoints, and (2) account for the complex interactions between occlusion, parallax, shading, and their view-dependent effects on scene-appearance.     

Pose-invariant face recognition using a 3D deformable model

The paper proposes a novel, pose-invariant face recognition system based on a deformable, generic 3D face model, that is a composite of: (1) an edge model, (2) a color region model and (3) a wireframe model for jointly describing the shape and important features of the face. The first two submodels are used for image analysis and the third mainly for face synthesis. In order to match the model to face images in arbitrary poses, the 3D model can be projected onto different 2D viewplanes based on rotation, translation and scale parameters, thereby generating multiple face-image templates (in different sizes and orientations). Face shape variations among people are taken into account by the deformation parameters of the model. Given an unknown face, its pose is estimated by model matching and the system synthesizes face images of known subjects in the same pose. The face is then classified as the subject whose synthesized image is most similar. The synthesized images are generated using a 3D face representation scheme which encodes the 3D shape and texture characteristics of the faces. This face representation is automatically derived from training face images of the subject. Experimental results show that the method is capable of determining pose and recognizing faces accurately over a wide range of poses and with naturally varying lighting conditions. Recognition rates of 92.3% have been achieved by the method with 10 training face images per person.     

A Long Memory Pattern Modelling and Recognition System for Financial Time-Series Forecasting

In this paper, the concept of a long memory system for forecasting is developed. Pattern modelling and recognition systems are introduced as local approximation tools for forecasting. Such systems are used for matching the current state of the time-series with past states to make a forecast. In the past, this system has been successfully used for forecasting the Santa Fe competition data. In this paper, we forecast the financial indices of six different countries, and compare the results with neural networks on five different error measures. The results show that pattern recognition-based approaches in time-series forecasting are highly accurate, and that these are able to match the performance of advanced methods such as neural networks. Received: 2 April 1998?Received in revised form: 1 February 1999?Accepted: 16 February 1999     

深度特征融合的非刚性三维形状局部匹配

针对海量、异构三维形状匹配与智能检索技术的需求,提出了一种基于级联卷积神经网络(F-PointCNN)深度特征融合的三维形状局部匹配方法.首先,采用特征袋模型,提出几何图像表示方法,该几何图像不仅能够有效区分同类异构的非刚性三维模型,而且能够揭示大尺度不完整三维模型的结构相似性.其次,构建级联卷积神经网络学习框架F-P...     

On the shape of a set of points and lines in the plane

Detailed geometric models of the real world are in increasing demand. LiDAR data is appropriate to reconstruct urban models. In urban scenes, the individual surfaces can be reconstructed and connected to form the scene geometry. There are various methods for reconstructing the free‐form shape of a point sample on a single surface. However, these methods do not take the context of the surface into account. We present the guided α‐shape: an extension of the well known α‐shape that uses lines (guides) to indicate preferred locations for the boundary of the shape. The guided α‐shape uses (parts of) these lines as boundary where the points suggest that this is appropriate. We prove that the guided α‐shape can be constructed in O((n + m) log (n + m)) time, from an input of n points and m guides. We apply guided α‐shapes to urban reconstruction from LiDAR, where neighboring surfaces can be connected conveniently along their intersection lines into adjacent surfaces of a 3D model. We analyze guided α‐shapes of both synthetic and real data and show they are consistently better than α‐shapes for this application.     

An online composite graphics recognition approach based on matching of spatial relation graphs

A spatial relation graph (SRG) and its partial matching method are proposed for online composite graphics representation and recognition. The SRG-based approach emphasizes three characteristics of online graphics recognition: partial, structural, and independent of stroke order and stroke number. A constrained partial permutation strategy is also proposed to reduce the computational cost of matching two SRGs, which is originally an NP-complete problem as is graph isomorphism. Experimental results show that our proposed SRG-based approach is both efficient and effective for online composite graphics recognition in our sketch-based graphics input system - SmartSketchpad.Received: 13 March 2003, Accepted: 13 March 2004, Published online: 1 June 2004     

Automatic construction of 2D shape models

A procedure for automated 2D shape model design is presented. The system is given a set of training example shapes defined by contour point coordinates. The shapes are automatically aligned using Procrustes analysis and clustered to obtain cluster prototypes (typical objects) and statistical information about intracluster shape variation. One difference from previous methods is that the training set is first automatically clustered and shapes considered to be outliers are discarded. In this way, cluster prototypes are not distorted by outliers. A second difference is in the manner in which registered sets of points are extracted from each shape contour. We propose a flexible point matching technique that takes into account both pose/scale differences and nonlinear shape differences. The matching method is independent of the objects' initial relative position/scale and does not require any manually tuned parameters. Our shape model design method was used to learn 11 different shapes from contours that were manually traced in MR brain images. The resulting model was then employed to segment several MR brain images that were not included in the shape-training set. A quantitative analysis of our shape registration approach, within the main cluster of each structure, demonstrated results that compare very well to those achieved by manual registration; achieving an average registration error of about 1 pixel. Our approach can serve as a fully automated substitute to the tedious and time-consuming manual 2D shape registration and analysis     

A multiview 3D modeling system based on stereo vision techniques

This paper introduces a stereo vision system to automatically generate 3D models of real objects. 3D model generation is based on the merging of multiview range images obtained from a digital stereo camera. Stereo images obtained from the camera are rectified, and a correlation-based stereo matching technique reconstructs range images from them. A turntable stage is also employed to obtain multiple range images of the objects. To register range images into a common coordinate system automatically, we introduce and calibrate a turntable coordinate system with respect to the camera coordinate system. After the registration of multiview range images, a 3D model is reconstructed using a volumetric integration technique. Error analysis on turntable calibration and 3D model reconstruction shows the accuracy of our 3D modeling system.Received: 2 August 2003, Accepted: 20 September 2004, Published online: 25 February 2005 Correspondence to: S.Y. Park     

A systematic approach for 2D-image to 3D-range registration in urban environments

The photorealistic modeling of large-scale objects, such as urban scenes, requires the combination of range sensing technology and digital photography. In this paper, we attack the key problem of camera pose estimation, in an automatic and efficient way. First, the camera orientation is recovered by matching vanishing points (extracted from 2D images) with 3D directions (derived from a 3D range model). Then, a hypothesis-and-test algorithm computes the camera positions with respect to the 3D range model by matching corresponding 2D and 3D linear features. The camera positions are further optimized by minimizing a line-to-line distance. The advantage of our method over earlier work has to do with the fact that we do not need to rely on extracted planar facades, or other higher-order features; we are utilizing low-level linear features. That makes this method more general, robust, and efficient. We have also developed a user-interface for allowing users to accurately texture-map 2D images onto 3D range models at interactive rates. We have tested our system in a large variety of urban scenes.