Techniques used and open challenges to the analysis,indexing and retrieval of digital video

Video in digital format is now commonplace and widespread in both professional use, and in domestic consumer products from camcorders to mobile phones. Video content is growing in volume and while we can capture, compress, store, transmit and display video with great facility, editing videos and manipulating them based on their content is still a non-trivial activity. In this paper, we give a brief review of the state of the art of video analysis, indexing and retrieval and we point to research directions which we think are promising and could make searching and browsing of video archives based on video content, as easy as searching and browsing (text) web pages. We conclude the paper with a list of grand challenges for researchers working in the area.     

Retrieving information from hypertext: issues and problems

One of the most important developments in information science and in information systems in recent years has been the emergence of hypertext as a way of organizing information. Hypertext uses a browsing rather than a searching strategy and hypertext systems have found applications in a number of diverse areas. One of the strengths of using a hypertext system is the freedom it gives to users in the choice of information they wish to see, but this is also one of its drawbacks. This paper examines the effectiveness of hypertext as a way of retrieving information and reviews conventional information retrieval techniques which are the alternative strategy. Previous attempts at combining information retrieval and hypertext are considered and a prototype system outlined which has been developed to generate guided tours for users, to direct them through a hypertext to information they have requested. Finally, how adding this kind of intelligent guidance to a hypertext system would affect its usability as an information system is discussed.     

Symbiosis between the TRECVid benchmark and video libraries at the Netherlands Institute for Sound and Vision

Audiovisual archives are investing in large-scale digitization efforts of their analogue holdings and, in parallel, ingesting an ever-increasing amount of born-digital files in their digital storage facilities. Digitization opens up new access paradigms and boosted re-use of audiovisual content. Query-log analyses show the shortcomings of manual annotation, therefore archives are complementing these annotations by developing novel search engines that automatically extract information from both audio and the visual tracks. Over the past few years, the TRECVid benchmark has developed a novel relationship with the Netherlands Institute of Sound and Vision (NISV) which goes beyond the NISV just providing data and use cases to TRECVid. Prototype and demonstrator systems developed as part of TRECVid are set to become a key driver in improving the quality of search engines at the NISV and will ultimately help other audiovisual archives to offer more efficient and more fine-grained access to their collections. This paper reports the experiences of NISV in leveraging the activities of the TRECVid benchmark.     

Automatic summarization of rushes video using bipartite graphs

In this paper we present a new approach for automatic summarization of rushes, or unstructured video. Our approach is composed of three major steps. First, based on shot and sub-shot segmentations, we filter sub-shots with low information content not likely to be useful in a summary. Second, a method using maximal matching in a bipartite graph is adapted to measure similarity between the remaining shots and to minimize inter-shot redundancy by removing repetitive retake shots common in rushes video. Finally, the presence of faces and motion intensity are characterised in each sub-shot. A measure of how representative the sub-shot is in the context of the overall video is then proposed. Video summaries composed of keyframe slideshows are then generated. In order to evaluate the effectiveness of this approach we re-run the evaluation carried out by TRECVid, using the same dataset and evaluation metrics used in the TRECVid video summarization task in 2007 but with our own assessors. Results show that our approach leads to a significant improvement on our own work in terms of the fraction of the TRECVid summary ground truth included and is competitive with the best of other approaches in TRECVid 2007.     

Video digital libraries: contributive and decentralised

Technology usage is changing rapidly and is becoming a more mobile, more social and more multimedia-based experience. This is especially true in the area of content creation where mobile social applications used by crowds of people are challenging traditional ways of creating and distributing content, especially for applications like news dissemination. Libraries have traditionally functioned as repositories where the information content of a society is analysed, curated, organised and stored, acting as a permanent record of what is to be remembered from a society. How can this function be achieved by present-day libraries attempting to cope with mobile, social, multimedia content who’s nature and utility of which change the type of information we wish to curate and store? This information is both dynamic and organic, posing challenges to the more fixed models of information in digital libraries. In this article we describe two digital library systems that archive video content from the sports domain, and which support user annotations and merging of diverse information sources in an integrated way. We report on analysis of the deployment of these two systems and highlight how they extend the traditional role of a (digital) library.     

Collaborative video searching on a tabletop

Almost all system and application design for multimedia systems is based around a single user working in isolation to perform some task yet much of the work for which we use computers to help us, is based on working collaboratively with colleagues. Groupware systems do support user collaboration but typically this is supported through software and users still physically work independently. Tabletop systems, such as the DiamondTouch from MERL, are interface devices which support direct user collaboration on a tabletop. When a tabletop is used as the interface for a multimedia system, such as a video search system, then this kind of direct collaboration raises many questions for system design. In this paper we present a tabletop system for supporting a pair of users in a video search task and we evaluate the system not only in terms of search performance but also in terms of user–user interaction and how different user personalities within each pair of searchers impacts search performance and user interaction. Incorporating the user into the system evaluation as we have done here reveals several interesting results and has important ramifications for the design of a multimedia search system.