NeTra: A toolbox for navigating large image databases

We present here an implementation of NeTra, a prototype image retrieval system that uses color, texture, shape and spatial location information in segmented image regions to search and retrieve similar regions from the database. A distinguishing aspect of this system is its incorporation of a robust automated image segmentation algorithm that allows object- or region-based search. Image segmentation significantly improves the quality of image retrieval when images contain multiple complex objects. Images are segmented into homogeneous regions at the time of ingest into the database, and image attributes that represent each of these regions are computed. In addition to image segmentation, other important components of the system include an efficient color representation, and indexing of color, texture, and shape features for fast search and retrieval. This representation allows the user to compose interesting queries such as “retrieve all images that contain regions that have the color of object A, texture of object B, shape of object C, and lie in the upper of the image”, where the individual objects could be regions belonging to different images. A Java-based web implementation of NeTra is available at http://vivaldi.ece.ucsb.edu/Netra.     

Combining multi-visual features for efficient indexing in a large image database

The optimized distance-based access methods currently available for multidimensional indexing in multimedia databases have been developed based on two major assumptions: a suitable distance function is known a priori and the dimensionality of the image features is low. It is not trivial to define a distance function that best mimics human visual perception regarding image similarity measurements. Reducing high-dimensional features in images using the popular principle component analysis (PCA) might not always be possible due to the non-linear correlations that may be present in the feature vectors. We propose in this paper a fast and robust hybrid method for non-linear dimensions reduction of composite image features for indexing in large image database. This method incorporates both the PCA and non-linear neural network techniques to reduce the dimensions of feature vectors so that an optimized access method can be applied. To incorporate human visual perception into our system, we also conducted experiments that involved a number of subjects classifying images into different classes for neural network training. We demonstrate that not only can our neural network system reduce the dimensions of the feature vectors, but that the reduced dimensional feature vectors can also be mapped to an optimized access method for fast and accurate indexing. Received 11 June 1998 / Accepted 25 July 2000 Published online: 13 February 2001     

Automated delineation of individual tree crowns in high spatial resolution aerial images by multiple-scale analysis

This paper presents an automatic multiple-scale algorithm for delineation of individual tree crowns in high spatial resolution infrared colour aerial images. The tree crown contours were identified as zero-crossings, with convex grey-level curvature, which were computed on the intensity image for each image scale. A modified centre of curvature was estimated for every edge segment pixel. For each segment, these centre points formed a swarm which was modelled as a primal sketch using an ellipse extended with the mean circle of curvature. The model described the region of the derived tree crown based on the edge segment at the current scale. The sketch was rescaled with a significance value and accumulated for a scale interval. In the accumulated sketch, a tree crown segment was grown, starting at local peaks, under the condition that it was inside the area of healthy vegetation in the aerial image and did not trespass into a neighbouring crown segment. The method was evaluated by comparison with manual delineation and with ground truth on 43 randomly selected sample plots. It was concluded that the performance of the method is almost equivalent to visual interpretation. On the average, seven out of ten tree crowns were the same. Furthermore, ground truth indicated a large number of hidden trees. The proposed technique could be used as a basic tool in forest surveys. Received: 24 June 1997 / Accepted: 28 April 1998     

Silhouette-based occluded object recognition through curvature scale space

A complete and practical system for occluded object recognition has been developed which is very robust with respect to noise and local deformations of shape (due to weak perspective distortion, segmentation errors and non-rigid material) as well as scale, position and orientation changes of the objects. The system has been tested on a wide variety of free-form 3D objects. An industrial application is envisaged where a fixed camera and a light-box are utilized to obtain images. Within the constraints of the system, every rigid 3D object can be modeled by a limited number of classes of 2D contours corresponding to the object's resting positions on the light-box. The contours in each class are related to each other by a 2D similarity transformation. The Curvature Scale Space technique [26, 28] is then used to obtain a novel multi-scale segmentation of the image and the model contours. Object indexing [16, 32, 36] is used to narrow down the search space. An efficient local matching algorithm is utilized to select the best matching models. Received: 5 August 1996 / Accepted: 19 March 1997     

A system for human identification from X-ray dental radiographs

Forensic odontology is the branch of forensics that deals with human identification based on dental features. In this paper, we present a system for automating that process by identifying people from dental X-ray images. Given a dental image of a postmortem (PM), the proposed system retrieves the best matches from an antemortem (AM) database. The system automatically segments dental X-ray images into individual teeth and extracts the contour of each tooth. Features are extracted from each tooth and are used for retrieval. We developed a new method for teeth separation based on integral projection. We also developed a new method for representing and matching teeth contours using signature vectors obtained at salient points on the contours of the teeth. During retrieval, the AM radiographs that have signatures closer to the PM are found and presented to the user. Matching scores are generated based on the distance between the signature vectors of AM and PM teeth. Experimental results on a small database of dental radiographs are encouraging.     

Fast image retrieval using color-spatial information

In this paper, we present an image retrieval system that employs both the color and spatial information of images to facilitate the retrieval process. The basic unit used in our technique is a single-colored cluster, which bounds a homogeneous region of that color in an image. Two clusters from two images are similar if they are of the same color and overlap in the image space. The number of clusters that can be extracted from an image can be very large, and it affects the accuracy of retrieval. We study the effect of the number of clusters on retrieval effectiveness to determine an appropriate value for “optimal' performance. To facilitate efficient retrieval, we also propose a multi-tier indexing mechanism called the Sequenced Multi-Attribute Tree (SMAT). We implemented a two-tier SMAT, where the first layer is used to prune away clusters that are of different colors, while the second layer discriminates clusters of different spatial locality. We conducted an experimental study on an image database consisting of 12,000 images. Our results show the effectiveness of the proposed color-spatial approach, and the efficiency of the proposed indexing mechanism. Received August 1, 1997 / Accepted December 9, 1997     

Simulation of postoperative 3D facial morphology using a physics-based head model

We propose a prototype of a facial surgery simulation system for surgical planning and the prediction of facial deformation. We use a physics-based human head model. Our head model has a 3D hierarchical structure that consists of soft tissue and the skull, constructed from the exact 3D CT patient data. Anatomic points measured on X-ray images from both frontal and side views are used to fire the model to the patient's head. The purposes of this research is to analyze the relationship between changes of mandibular position and facial morphology after orthognathic surgery, and to simulate the exact postoperative 3D facial shape. In the experiment, we used our model to predict the facial shape after surgery for patients with mandibular prognathism. Comparing the simulation results and the actual facial images after the surgery shows that the proposed method is practical.     

Registration of technical drawings and calibrated images for industrial augmented reality

Despite tremendous progress in 3D modelling technology, most sites in traditional industries do not have a computer model of their facilities at their disposal. In these industries, 2D technical drawings are typically the most commonly used documents. In many cases, a database of fully calibrated and oriented photogrammetric images of parts of the plant is also available. These images are often used for metric measurement and 3D as-built modelling. For planning revamps and maintenance, it is necessary to use industrial drawings as well as images and 3D models represented in a common “world” coordinate system. This paper proposes a method for full integration of technical drawings, calibrated images and as-built 3D models. A new algorithm is developed in order to use only a few correspondences between points on a technical drawing and multiple images to estimate a metric planar transformation between the drawing and the world coordinate system. The paper describes the mathematical relationship between this transformation and the set of homographies needed for merging the technical drawing with all the calibrated images. The method is implemented and fully integrated into an industrial software we developed for 3D as-built reconstruction. We present examples of a real application, in which the method is successfully applied to create an augmented reality representation of a waste water plant. Accepted: 13 August 2001     

Multipass hierarchical stereo matching for generation of digital terrain models from aerial images

This paper presents a new multi-pass hierarchical stereo-matching approach for generation of digital terrain models (DTMs) from two overlapping aerial images. Our method consists of multiple passes which compute stereo matches with a coarse-to-fine and sparse-to-dense paradigm. An image pyramid is generated and used in the hierarchical stereo matching. Within each pass, the DTM is refined by using the image pyramid from the coarse to the fine level. At the coarsest level of the first pass, a global stereo-matching technique, the intra-/inter-scanline matching method, is used to generate a good initial DTM for the subsequent stereo matching. Thereafter, hierarchical block matching is applied to image locations where features are detected to refine the DTM incrementally. In the first pass, only the feature points near salient edge segments are considered in block matching. In the second pass, all the feature points are considered, and the DTM obtained from the first pass is used as the initial condition for local searching. For the passes after the second pass, 3D interactive manual editing can be incorporated into the automatic DTM refinement process whenever necessary. Experimental results have shown that our method can successfully provide accurate DTM from aerial images. The success of our approach and system has also been demonstrated with a flight simulation software. Received: 4 November 1996 / Accepted: 20 October 1997     

Integrating symbolic images into a multimedia database system using classification and abstraction approaches

Symbolic images are composed of a finite set of symbols that have a semantic meaning. Examples of symbolic images include maps (where the semantic meaning of the symbols is given in the legend), engineering drawings, and floor plans. Two approaches for supporting queries on symbolic-image databases that are based on image content are studied. The classification approach preprocesses all symbolic images and attaches a semantic classification and an associated certainty factor to each object that it finds in the image. The abstraction approach describes each object in the symbolic image by using a vector consisting of the values of some of its features (e.g., shape, genus, etc.). The approaches differ in the way in which responses to queries are computed. In the classification approach, images are retrieved on the basis of whether or not they contain objects that have the same classification as the objects in the query. On the other hand, in the abstraction approach, retrieval is on the basis of similarity of feature vector values of these objects. Methods of integrating these two approaches into a relational multimedia database management system so that symbolic images can be stored and retrieved based on their content are described. Schema definitions and indices that support query specifications involving spatial as well as contextual constraints are presented. Spatial constraints may be based on both locational information (e.g., distance) and relational information (e.g., north of). Different strategies for image retrieval for a number of typical queries using these approaches are described. Estimated costs are derived for these strategies. Results are reported of a comparative study of the two approaches in terms of image insertion time, storage space, retrieval accuracy, and retrieval time. Received June 12, 1998 / Accepted October 13, 1998     

Spectral covariance and fuzzy regions for image indexing

To improve the discrimination power of color-indexing techniques, we encode a minimal amount of spatial information in the index. We tesselate each image with five partially overlapping, fuzzy regions. In the index, for each region in an image, we store its average color and the covariance matrix of the color distribution. A similiarity function of these color features is used to match query images with images in the database. In addition, we propose two measures to evaluate the performance of image-indexing techniques. We present experimental results using an image database which contains more than 11,600 color images.     

Using vanishing points for camera calibration and coarse 3D reconstruction from a single image

In this paper, we show how to calibrate a camera and to recover the geometry and the photometry (textures) of objects from a single image. The aim of this work is to make it possible walkthrough and augment reality in a 3D model reconstructed from a single image. The calibration step does not need any calibration target and makes only four assumptions: (1) the single image contains at least two vanishing points, (2) the length (in 3D space) of one line segment (for determining the translation vector) in the image is known, (3) the principle point is the center of the image, and (4) the aspect ratio is fixed by the user. Each vanishing point is determined from a set of parallel lines. These vanishing points help determine a 3D world coordinate system R _o. After having computed the focal length, the rotation matrix and the translation vector are evaluated in turn for describing the rigid motion between R _o and the camera coordinate system R _c. Next, the reconstruction step consists in placing, rotating, scaling, and translating a rectangular 3D box that must fit at best with the potential objects within the scene as seen through the single image. With each face of a rectangular box, a texture that may contain holes due to invisible parts of certain objects is assigned. We show how the textures are extracted and how these holes are located and filled. Our method has been applied to various real images (pictures scanned from books, photographs) and synthetic images.     

Image retrieval using efficient local-area matching

We present an efficient and accurate method for retrieving images based on color similarity with a given query image or histogram. The method matches the query against parts of the image using histogram intersection. Efficient searching for the best matching subimage is done by pruning the set of subimages using upper bound estimates. The method is fast, has high precision and recall and also allows queries based on the positions of one or more objects in the database image. Experimental results showing the efficiency of the proposed search method, and high precision and recall of retrieval are presented. Received: 20 January 1997 / Accepted: 5 January 1998     

The reconstruction of dynamic 3D structure of biological objects using stereo microscope images

In this paper, we address the analysis of 3D shape and shape change in non-rigid biological objects imaged via a stereo light microscope. We propose an integrated approach for the reconstruction of 3D structure and the motion analysis for images in which only a few informative features are available. The key components of this framework are: 1) image registration using a correlation-based approach, 2) region-of-interest extraction using motion-based segmentation, and 3) stereo and motion analysis using a cooperative spatial and temporal matching process. We describe these three stages of processing and illustrate the efficacy of the proposed approach using real images of a live frog's ventricle. The reconstructed dynamic 3D structure of the ventricle is demonstrated in our experimental results, and it agrees qualitatively with the observed images of the ventricle.     

Towards the recovery of extrinsic camera parameters from video records of sewer surveys

The aim of the work reported here is the recovery, from a single image taken inside a roughly cylindrical brick sewer pipe of diameter up to one meter, of the pose of the camera relative to the central axis of the pipe. It is shown that the vanishing point associated with the longitudinal mortar lines carries valuable information about the pose. A method for the automatic detection of this point is presented and used to analyse the camera rotations underlying a number of sewer survey videos. It is similarly shown how the angles between the images of the longitudinal lines can be used to recover information about camera pose. The techniques might form an active part of a more comprehensive image understanding system recovering the three-dimensional shape of a surveyed pipe from survey videos and/or be used as an experimental tool during the design of such a system. Received: 24 June 1997 / Accepted: 17 March 1998     

Tracking a person with pre-recorded image database and a pan, tilt, and zoom camera

Abstract. This paper proposes a novel tracking strategy that can robustly track a person or other object within a fixed environment using a pan, tilt, and zoom camera with the help of a pre-recorded image database. We define a set of camera states which is sufficient to survey the environment for the target. Background images for these camera states are stored as an image database. During tracking, camera movements are restricted to these states. Tracking and segmentation are simplified, as each tracking image can be compared with the corresponding pre-recorded background image. Received: 26 August 1999 / Accepted: 22 February 2000