Multimodal Video Indexing: A Review of the State-of-the-art

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming at automating this time and resource consuming process. Good reviews on single modality based video indexing have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. Therefore, instead of separately treating the different information sources involved, and their specific algorithms, we focus on the similarities and differences between the modalities. To that end we put forward a unifying and multimodal framework, which views a video document from the perspective of its author. This framework forms the guiding principle for identifying index types, for which automatic methods are found in literature. It furthermore forms the basis for categorizing these different methods.     

A semi-interactive panorama based 3D reconstruction framework for indoor scenes

We present a semi-interactive method for 3D reconstruction specialized for indoor scenes which combines computer vision techniques with efficient interaction. We use panoramas, popularly used for visualization of indoor scenes, but clearly not able to show depth, for their great field of view, as the starting point. Exploiting user defined knowledge, in term of a rough sketch of orthogonality and parallelism in scenes, we design smart interaction techniques to semi-automatically reconstruct a scene from coarse to fine level. The framework is flexible and efficient. Users can build a coarse walls-and-floor textured model in five mouse clicks, or a detailed model showing all furniture in a couple of minutes interaction. We show results of reconstruction on four different scenes. The accuracy of the reconstructed models is quite high, around 1% error at full room scale. Thus, our framework is a good choice for applications requiring accuracy as well as application requiring a 3D impression of the scene.     

Digitized circular arcs: characterization and parameter estimation

The digitization of a circular arc causes an inherent loss of geometrical information. Arcs with slightly different local curvature or position may lead to exactly the same digital pattern. In this paper the authors give a characterization of all centers and radii of circular arcs yielding the same digitization pattern. The radius of the arcs varies over the set. However, only one curvature or radius estimate can be assigned to the digital pattern. The authors derive an optimal estimator and give expressions for the bound on the precision of estimation. This bound due to digitization is the deterministic equivalent of the Cramer/Rao bound known from parameter estimation theory. Consider the estimation of the local curvature and local radius of a smooth object. Typically such parameters are estimated by moving a window along the digital boundary. Methods in literature show a poor precision in estimating curvature values, relative errors of over 40% are often found. From the definition of curvature it follows that locally the curve can be considered a circular arc and hence the method presented in this paper can be applied to the pattern in the window giving estimates with optimal precision and a measure for the remaining error. On the practical side the authors present examples of the residual error due to the discrete grid. The estimation of the radius or curvature of a circular arc at random position with an estimation window containing 10 points (coded with nine Freemancodes) has a relative deviation exceeding 2%. For a full disk the deviation is below 1% when the radius r exceeds four grid units. The presented method is particularly useful for problems where some prior knowledge on the distribution of radii is known and where there is a noise-free sampling     

Content based internet access to paper documents

When archives of paper documents are to be accessed via the Internet, the implicit hypertext structure of the original documents should be employed. In this paper we study the different hypertext structures one encounters in a document. Methods for analyzing paper documents to find these structures are presented. The structures also form the basis for the presentation of the content of the document to the user. Results are presented. Received October 13, 1998 / Revised February 15, 1999     

Learning Social Tag Relevance by Neighbor Voting

Social image analysis and retrieval is important for helping people organize and access the increasing amount of user tagged multimedia. Since user tagging is known to be uncontrolled, ambiguous, and overly personalized, a fundamental problem is how to interpret the relevance of a user-contributed tag with respect to the visual content the tag is describing. Intuitively, if different persons label visually similar images using the same tags, these tags are likely to reflect objective aspects of the visual content. Starting from this intuition, we propose in this paper a neighbor voting algorithm which accurately and efficiently learns tag relevance by accumulating votes from visual neighbors. Under a set of well-defined and realistic assumptions, we prove that our algorithm is a good tag relevance measurement for both image ranking and tag ranking. Three experiments on 3.5 million Flickr photos demonstrate the general applicability of our algorithm in both social image retrieval and image tag suggestion. Our tag relevance learning algorithm substantially improves upon baselines for all the experiments. The results suggest that the proposed algorithm is promising for real-world applications.