Multipass hierarchical stereo matching for generation of digital terrain models from aerial images

This paper presents a new multi-pass hierarchical stereo-matching approach for generation of digital terrain models (DTMs) from two overlapping aerial images. Our method consists of multiple passes which compute stereo matches with a coarse-to-fine and sparse-to-dense paradigm. An image pyramid is generated and used in the hierarchical stereo matching. Within each pass, the DTM is refined by using the image pyramid from the coarse to the fine level. At the coarsest level of the first pass, a global stereo-matching technique, the intra-/inter-scanline matching method, is used to generate a good initial DTM for the subsequent stereo matching. Thereafter, hierarchical block matching is applied to image locations where features are detected to refine the DTM incrementally. In the first pass, only the feature points near salient edge segments are considered in block matching. In the second pass, all the feature points are considered, and the DTM obtained from the first pass is used as the initial condition for local searching. For the passes after the second pass, 3D interactive manual editing can be incorporated into the automatic DTM refinement process whenever necessary. Experimental results have shown that our method can successfully provide accurate DTM from aerial images. The success of our approach and system has also been demonstrated with a flight simulation software. Received: 4 November 1996 / Accepted: 20 October 1997     

Computer-assisted coloring by matching line drawings

An approach to automatically color line drawings based on feature matching is proposed. The motivation is that coloring 2D animation is still a labor-intensive process in current cartoon film production. The objective of our work is to investigate how to automatically color an image in a cartoon sequence on the basis of the previous frame. Our method first establishes the matching relationship of two images, after which it automatically paints one of them with the color information of the other using a region-matching algorithm. The region-matching algorithm is based on feature correspondences. The results show that the proposed algorithm can straightforwardly and robustly realize our objective and has a promising future for our next step to further automate conventional animation.     

The reconstruction of dynamic 3D structure of biological objects using stereo microscope images

In this paper, we address the analysis of 3D shape and shape change in non-rigid biological objects imaged via a stereo light microscope. We propose an integrated approach for the reconstruction of 3D structure and the motion analysis for images in which only a few informative features are available. The key components of this framework are: 1) image registration using a correlation-based approach, 2) region-of-interest extraction using motion-based segmentation, and 3) stereo and motion analysis using a cooperative spatial and temporal matching process. We describe these three stages of processing and illustrate the efficacy of the proposed approach using real images of a live frog's ventricle. The reconstructed dynamic 3D structure of the ventricle is demonstrated in our experimental results, and it agrees qualitatively with the observed images of the ventricle.     

Silhouette-based occluded object recognition through curvature scale space

A complete and practical system for occluded object recognition has been developed which is very robust with respect to noise and local deformations of shape (due to weak perspective distortion, segmentation errors and non-rigid material) as well as scale, position and orientation changes of the objects. The system has been tested on a wide variety of free-form 3D objects. An industrial application is envisaged where a fixed camera and a light-box are utilized to obtain images. Within the constraints of the system, every rigid 3D object can be modeled by a limited number of classes of 2D contours corresponding to the object's resting positions on the light-box. The contours in each class are related to each other by a 2D similarity transformation. The Curvature Scale Space technique [26, 28] is then used to obtain a novel multi-scale segmentation of the image and the model contours. Object indexing [16, 32, 36] is used to narrow down the search space. An efficient local matching algorithm is utilized to select the best matching models. Received: 5 August 1996 / Accepted: 19 March 1997     

Analyzing scenery images by monotonic tree

Content-based image retrieval (CBIR) has been an active research area in the last ten years, and a variety of techniques have been developed. However, retrieving images on the basis of low-level features has proven unsatisfactory, and new techniques are needed to support high-level queries. Research efforts are needed to bridge the gap between high-level semantics and low-level features. In this paper, we present a novel approach to support semantics-based image retrieval. Our approach is based on the monotonic tree, a derivation of the contour tree for use with discrete data. The structural elements of an image are modeled as branches (or subtrees) of the monotonic tree. These structural elements are classified and clustered on the basis of such properties as color, spatial location, harshness and shape. Each cluster corresponds to some semantic feature. This scheme is applied to the analysis and retrieval of scenery images. Comparisons of experimental results of this approach with conventional techniques using low-level features demonstrate the effectiveness of our approach.     

Direct linear sub-pixel correlation by incorporation of neighbor pixels' information and robust estimation of window transformation

Standard methods for sub-pixel matching are iterative and nonlinear; they are also sensitive to false initialization and window deformation. In this paper, we present a linear method that incorporates information from neighboring pixels. Two algorithms are presented: one ‘fast’ and one ‘robust’. They both start from an initial rough estimate of the matching. The fast one is suitable for pairs of images requiring negligible window deformation. The robust method is slower but more general and more precise. It eliminates false matches in the initialization by using robust estimation of the local affine deformation. The first algorithm attains an accuracy of 0.05 pixels for interest points and 0.06 for random points in the translational case. For the general case, if the deformation is small, the second method gives an accuracy of 0.05 pixels; while for large deformation, it gives an accuracy of about 0.06 pixels for points of interest and 0.10 pixels for random points. They are very few false matches in all cases, even if there are many in the initialization. Received: 24 July 1997 / Accepted: 4 December 1997     

Global methods for stroke segmentation

Two methods for stroke segmentation from a global point of view are presented and compared. One is based on thinning methods and the other is based on contour curve fitting. For both cases an input image is binarized. For the former, Hilditch's method is used, then crossing points are sought, around which a domain is constructed. Outside the domain, a set of line segments are identified. These lines are connected and approximated by cubic B-spline curves. Smoothly connected lines are selected as segmented curves. This method works well for a limited class of crossing lines, which are shown experimentally. In the latter, a contour line is approximated by cubic B-spline curve, along which curvature is measured. According to the extreme points of the curvature graph, the contour line is segmented, based on which the line segment is obtained. Experimental results are shown for some difficult cases. Received October 31, 1998 / Revised January 12, 1999     

Shape-based word recognition

We describe a process of word recognition that has high tolerance for poor image quality, tunability to the lexical content of the documents to which it is applied, and high speed of operation. This process relies on the transformation of text images into character shape codes, and on special lexica that contain information on the shape of words. We rely on the structure of English and the high efficiency of mapping between shape codes and the characters in the words. Remaining ambiguity is reduced by template matching using exemplars derived from surrounding text, taking advantage of the local consistency of font, face and size as well as image quality. This paper describes the effects of lexical content, structure and processing on the performance of a word recognition engine. Word recognition performance is shown to be enhanced by the application of an appropriate lexicon. Recognition speed is shown to be essentially independent of the details of lexical content provided the intersection of the occurrences of words in the document and the lexicon is high. Word recognition accuracy is dependent on both intersection and specificity of the lexicon. Received May 1, 1998 / Revised October 20, 1998     

Image-based animation of facial expressions

We present a novel technique for creating realistic facial animations given a small number of real images and a few parameters for the in-between images. This scheme can also be used for reconstructing facial movies where the parameters can be automatically extracted from the images. The in-between images are produced without ever generating a three-dimensional model of the face. Since facial motion due to expressions are not well defined mathematically our approach is based on utilizing image patterns in facial motion. These patterns were revealed by an empirical study which analyzed and compared image motion patterns in facial expressions. The major contribution of this work is showing how parameterized “ideal” motion templates can generate facial movies for different people and different expressions, where the parameters are extracted automatically from the image sequence. To test the quality of the algorithm, image sequences (one of which was taken from a TV news broadcast) were reconstructed, yielding movies hardly distinguishable from the originals. Published online: 2 October 2002 Correspondence to: A. Tal Work has been supported in part by the Israeli Ministry of Industry and Trade, The MOST Consortium     

Query by video clip

Typical digital video search is based on queries involving a single shot. We generalize this problem by allowing queries that involve a video clip (say, a 10-s video segment). We propose two schemes: (i) retrieval based on key frames follows the traditional approach of identifying shots, computing key frames from a video, and then extracting image features around the key frames. For each key frame in the query, a similarity value (using color, texture, and motion) is obtained with respect to the key frames in the database video. Consecutive key frames in the database video that are highly similar to the query key frames are then used to generate the set of retrieved video clips. (ii) In retrieval using sub-sampled frames, we uniformly sub-sample the query clip as well as the database video. Retrieval is based on matching color and texture features of the sub-sampled frames. Initial experiments on two video databases (basketball video with approximately 16,000 frames and a CNN news video with approximately 20,000 frames) show promising results. Additional experiments using segments from one basketball video as query and a different basketball video as the database show the effectiveness of feature representation and matching schemes.     

A new and fast approach for DPIV using an incompressible affine flow model

Traditional digital particle image velocimetry (DPIV) methods are previously based on area-correlation. Though proven to be very time-consuming and error prone, it has been widely adopted because it is conceptually simple, and easy to implement, and also because there are few alternatives. This paper provides a non-correlative, conceptually new, fast and efficient approach for DPIV which takes the nature of flow into consideration. An incompressible affine flow model (IAFM) is introduced to describe a flow that incorporates rational constraint directly into the computation. This IAFM, combining with a modified optical flow method – named total optical flow computation, provides a linear system solution to DPIV. Experimental results on real images demonstrate our method to be a very promising approach for DPIV. Received: 23 March 1998 / Accepted: 1 September 1999     

Automatic reconstruction of 3D human arm motion from a monocular image sequence

A model-based approach to reconstruction of 3D human arm motion from a monocular image sequence taken under orthographic projection is presented. The reconstruction is divided into two stages. First, a 2D shape model is used to track the arm silhouettes and second-order curves are used to model the arm based on an iteratively reweighted least square method. As a result, 2D stick figures are extracted. In the second stage, the stick figures are backprojected into the scene. 3D postures are reconstructed using the constraints of a 3D kinematic model of the human arm. The motion of the arm is then derived as a transition between the arm postures. Applications of these results are foreseen in the analysis of human motion patterns. Received: 26 January 1996 / Accepted: 17 July 1997     

Optic flow estimation by a Hopfield neural network using geometrical constraints

Sparse optic flow maps are general enough to obtain useful information about camera motion. Usually, correspondences among features over an image sequence are estimated by radiometric similarity. When the camera moves under known conditions, global geometrical constraints can be introduced in order to obtain a more robust estimation of the optic flow. In this paper, a method is proposed for the computation of a robust sparse optic flow (OF) which integrates the geometrical constraints induced by camera motion to verify the correspondences obtained by radiometric-similarity-based techniques. A raw OF map is estimated by matching features by correlation. The verification of the resulting correspondences is formulated as an optimization problem that is implemented on a Hopfield neural network (HNN). Additional constraints imposed in the energy function permit us to achieve a subpixel accuracy in the image locations of matched features. Convergence of the HNN is reached in a small enough number of iterations to make the proposed method suitable for real-time processing. It is shown that the proposed method is also suitable for identifying independently moving objects in front of a moving vehicle. Received: 26 December 1995 / Accepted: 20 February 1997     

Detecting textured objects using convex hull

In this paper, we present a methodology of locating 3D objects of known shapes from a single gray-scale image, in particular objects with rich textures on the surface. While traditional approaches identify objects by grouping and matching local features, we locate the object in the image using its convex hull, a high level feature not given much attention in the image using literature. A “direct line detection” algorithm is developed to detect line segments directly from the gray-scale image divided in small blocks. Lines are clustered and convex hull of a single or group of clusters is computed and edited to extract the 2D contour of the object. Successful experiments on rectangular boxes and cylinders show the effectiveness of the convex hull approach and its potential usage in industrial applications. Part of the work discussed in this paper was performed when both authors were affiliated with Symbol Technologies.     

Iterative model-based binarization algorithm for cheque images

Binarization of document images with poor contrast, strong noise, complex patterns, and variable modalities in the gray-scale histograms is a challenging problem. A new binarization algorithm has been developed to address this problem for personal cheque images. The main contribution of this approach is optimizing the binarization of a part of the document image that suffers from noise interference, referred to as the Target Sub-Image (TSI), using information easily extracted from another noise-free part of the same image, referred to as the Model Sub-Image (MSI). Simple spatial features extracted from MSI are used as a model for handwriting strokes. This model captures the underlying characteristics of the writing strokes, and is invariant to the handwriting style or content. This model is then utilized to guide the binarization in the TSI. Another contribution is a new technique for the structural analysis of document images, which we call “Wavelet Partial Reconstruction” (WPR). The algorithm was tested on 4,200 cheque images and the results show significant improvement in binarization quality in comparison with other well-established algorithms. Received: October 10, 2001 / Accepted: May 7, 2002 This research was supported in part by NCR and NSERC's industrial postgraduate scholarship No. 239464. A simplified version of this paper has been presented at ICDAR 2001 [3].     

Real-time multiple vehicle detection and tracking from a moving vehicle

Abstract. A real-time vision system has been developed that analyzes color videos taken from a forward-looking video camera in a car driving on a highway. The system uses a combination of color, edge, and motion information to recognize and track the road boundaries, lane markings and other vehicles on the road. Cars are recognized by matching templates that are cropped from the input data online and by detecting highway scene features and evaluating how they relate to each other. Cars are also detected by temporal differencing and by tracking motion parameters that are typical for cars. The system recognizes and tracks road boundaries and lane markings using a recursive least-squares filter. Experimental results demonstrate robust, real-time car detection and tracking over thousands of image frames. The data includes video taken under difficult visibility conditions. Received: 1 September 1998 / Accepted: 22 February 2000     

Image retrieval using efficient local-area matching

We present an efficient and accurate method for retrieving images based on color similarity with a given query image or histogram. The method matches the query against parts of the image using histogram intersection. Efficient searching for the best matching subimage is done by pruning the set of subimages using upper bound estimates. The method is fast, has high precision and recall and also allows queries based on the positions of one or more objects in the database image. Experimental results showing the efficiency of the proposed search method, and high precision and recall of retrieval are presented. Received: 20 January 1997 / Accepted: 5 January 1998     

Local model checking and protocol analysis

This paper describes a local model-checking algorithm for the alternation-free fragment of the modal mu-calculus that has been implemented in the Concurrency Factory and discusses its application to the analysis of a real-time communications protocol. The protocol considered is RETHER, a software-based, real-time Ethernet protocol developed at SUNY at Stony Brook. Its purpose is to provide guaranteed bandwidth and deterministic, periodic network access to multimedia applications over commodity Ethernet hardware. Our model-checking results show that (for a particular network configuration) RETHER makes good on its bandwidth guarantees to real-time nodes without exposing non-real-time nodes to the possibility of starvation. Our data also indicate that, in many cases, the state-exploration overhead of the local model checker is significantly smaller than the total amount that would result from a global analysis of the protocol. In the course of specifying and verifying RETHER, we also identified an alternative design of the protocol that warranted further study due to its potentially smaller run-time overhead in servicing requests for data transmission. Again, using local model checking, we showed that this alternative design also possesses the properties of interest. This observation points out one of the often-overlooked benefits of formal verification: by forcing designers to understand their designs rigorously and abstractly, these techniques often enable the designers to uncover interesting design alternatives.