基于MPEG-7的视频语义检索系统

设计和实现一个支持语义的分布式视频检索系统:"语寻"。该系统利用一个改进的视频语义处理工具(该工具基于IBM VideoAnnEx标注工具,并增加镜头语义图标注和自然语言处理的功能)对视频进行语义分析和标注,生成包含语义信息的MPEG-7描述文件,然后对视频的MPEG-7描述文件建立分布式索引,并同时分布式存储视频文件;系统提供丰富的Web查询接口,包括关键字语义扩展查询,语义图查询以及自然语句查询,当用户提交语义查询意图后,便能够迅速地检索到感兴趣的视频和片段,并且可以浏览点播;整个系统采用分布式架构,具备良好的可扩展性,并能够支持海量视频信息的索引和检索。     

An Annotation Engine for Supporting Video Database Population

Segmentation, video data modeling, and annotation are indispensable operations necessary for creating and populating a video database. To support such video databases, annotation data can be collected as metadata for the database and subsequently used for indexing and query evaluation. In this paper we describe the design and development of a video annotation engine, called Vane, intended to solve this problem as a domain-independent video annotation application.Using the Vane tool, the annotation of raw video data is achieved through metadata collection. This process, which is performed semi-automatically, produces tailored SGML documents whose purpose is to describe information about the video content. These documents constitute the metadatabase component of the video database. The video data model which has been developed for the metadata, is as open as possible for multiple domain-specific applications. The tool is currently in use to annotate a video archive comprised of educational and news video content.     

Rule-based spatiotemporal query processing for video databases

In our earlier work, we proposed an architecture for a Web-based video database management system (VDBMS) providing an integrated support for spatiotemporal and semantic queries. In this paper, we focus on the task of spatiotemporal query processing and also propose an SQL-like video query language that has the capability to handle a broad range of spatiotemporal queries. The language is rule-based in that it allows users to express spatial conditions in terms of Prolog-type predicates. Spatiotemporal query processing is carried out in three main stages: query recognition, query decomposition, and query execution.Received: 11 October 2001, Accepted: 3 October 2003, Published online: 12 December 2003Edited by: A. Buchmann Correspondence to: Özgür UlusoyThis work is supported by the Scientific and Research Council of Turkey (TÜBITAK) under Project Code 199E025. This work was done while the first author was at Bilkent University.     

一种基于半监督聚类的微视频标注方法

针对微视频语义标注问题,本文提出一种基于半监督聚类的微视频标注方法。文中从事件驱动的角度,以镜头事件为单位,用事件组来标注微视频。进一步构造半监督K-means聚类算法,优化目标函数,使得最终的聚类结果既体现类间的低耦合及类内的高聚合,又体现类内局部的数据分布密度。该聚类算法实现了诸如微视频等多属性异构数据的聚类,提高了微视频标注效果。实验结果表明本文所提微视频标注方法具有很强的语义表达能力,本文的聚类方法也具有很强的聚类准确度。     

A general framework for representing,reasoning and querying with annotated Semantic Web data

We describe a generic framework for representing and reasoning with annotated Semantic Web data, a task becoming more important with the recent increased amount of inconsistent and non-reliable meta-data on the Web. We formalise the annotated language, the corresponding deductive system and address the query answering problem. Previous contributions on specific RDF annotation domains are encompassed by our unified reasoning formalism as we show by instantiating it on (i) temporal, (ii) fuzzy, and (iii) provenance annotations. Moreover, we provide a generic method for combining multiple annotation domains allowing to represent, e.g., temporally-annotated fuzzy RDF. Furthermore, we address the development of a query language – AnQL – that is inspired by SPARQL, including several features of SPARQL 1.1 (subqueries, aggregates, assignment, solution modifiers) along with the formal definitions of their semantics.     

Heuristics for Optimizing Multi-Clip Queries in Video Databases

In this paper we address the multi-clip query optimization problem where a multi-clip query requests multiple video clips. We propose a new heuristics called Restricted Search Interval that maximizes clip sharing between queries and consequently reduces the network bandwidth of a video server for a multicast system. An adaptation of our heuristics for optimizing the response time of the query is also presented. The experimental results show that the suggested heuristics reduces the server workload by about 28% on the average in comparison to a classical heuristic approach.     

Annotations for streaming video on the Web: system design and usage studies

Streaming video on the World Wide Web is being widely deployed, and workplace training and distance education are key applications. The ability to annotate video on the Web can provide significant added value in these and other areas. Written and spoken annotations can provide `in context' personal notes and can enable asynchronous collaboration among groups of users. With annotations, users are no longer limited to viewing content passively on the Web, but are free to add and share commentary and links, thus transforming the Web into an interactive medium. We discuss design considerations in constructing a collaborative video annotation system, and we introduce our prototype, called MRAS. We present preliminary data on the use of Web-based annotations for personal note-taking and for sharing notes in a distance education scenario. Users showed a strong preference for MRAS over pen-and-paper for taking notes, despite taking longer to do so. They also indicated that they would make more comments and questions with MRAS than in a `live' situation, and that sharing added substantial value.     

基于时序概率超图模型的视频多语义标注

在基于语义的视频检索系统中,为了弥补视频底层特征与高层用户需求之间的差异,提出了时序概率超图模型。它将时间序列因素融入到模型的构建中,在此基础上提出了一种基于时序概率超图模型的视频多语义标注框架（TPH-VMLAF）。该框架结合视频时间相关性,通过使用基于时序概率超图的镜头多标签半监督分类学习算法对视频镜头进行多语义标注。标注过程中同时解决了已标注视频数据不足和多语义标注的问题。实验结果表明,该框架提高了标注的精确度,表现出了良好的性能。     

不一致数据库上带信任标记的查询结果

不一致数据无法正确反映现实世界,其上的查询结果内含错误或矛盾,而现有的很多不一致数据查询处理相关研究都存在信息丢失的问题.AQA(annotation based query answer)针对这一问题采用信任标签在属性级别上区分一致和不一致数据,避免了信息丢失.但AQA假设记录在依赖左边属性上的分量可信,且只针对函数依赖一种约束,具有应用局限性.在综合约束(函数依赖、包含依赖和域约束)范围内、不确定属性任意的情况下扩展了AQA,重新审视了AQA的数据模型及其上的查询代数,讨论了任意约束在查询结果上的蕴含约束计算问题.实验结果表明,扩展后的AQA非连接类查询的性能和普通的SQL基夺相同,连接查询经优化后性能接近普通SQL查询,但AQA不丢失信息与部分同类研究相比有很大优势.     

Hierarchical representation of video sequences for annotation

Video annotation is an important issue in video content management systems. Rapid growth of the digital video data has created a need for efficient and reasonable mechanisms that can ease the annotation process. In this paper, we propose a novel hierarchical clustering based system for video annotation. The proposed system generates a top-down hierarchy of the video streams using hierarchical k-means clustering. A tree-based structure is produced by dividing the video recursively into sub-groups, each of which consists of similar content. Based on the visual features, each node of the tree is partitioned into its children using k-means clustering. Each sub-group is then represented by its key frame, which is selected as the closest frame to the centroids of the corresponding cluster, and then can be displayed at the higher level of the hierarchy. The experiments show that very good hierarchical view of the video sequences can be created for annotation in terms of efficiency.     

3D-List: a data structure for efficient video query processing

A video query model based on the content of video and iconic indexing is proposed. We extend the notion of two-dimensional strings to three-dimensional strings (3D-Strings) for representing the spatial and temporal relationships among the symbols in both a video and a video query. The problem of video query processing is then transformed into a problem of three-dimensional pattern matching. To efficiently match the 3D-Strings, a data structure, called 3D-List, and its related algorithms are proposed. In this approach, the symbols of a video in the video database are retrieved from the video index and organized as a 3D-List according to the 3D-String of the video query. The related algorithms are then applied on the 3D-List to determine whether this video is an answer to the video query. Based on this approach, we have started a project called Vega. In this project, we have implemented a user friendly interface for specifying video queries, a video index tool for constructing the video index, and a video query processor based on the notion of 3D-List. Some experiments are also performed to show the efficiency and effectiveness of the proposed algorithms     

Integrated Video and Text for Content-based Access to Video Databases

This paper introduces a new approach to realize video databases. The approach consists of a VideoText data model based on free text annotations associated with logical video segments and a corresponding query language. Traditional database techniques are inadequate for exploiting queries on unstructured data such as video, supporting temporal queries, and ranking query results according to their relevance to the query. In this paper, we propose to use information retrieval techniques to provide such features and to extend the query language to accommodate interval queries that are particularly suited to video data. Algorithms are provided to show how user queries are evaluated. Finally, a generic and modular video database architecture which is based on VideoText data model is described.     

基于时空灰度序特征的视频片段定位算法

基于灰度序特征的视频片段定位算法是解决视频片段定位问题的典型算法.这类算法存在的不足是：特征的唯一性表示能力不够,使得在召回率较高的情况下,定位检索的精度下降得较快;二次多项式级的时间复杂度使得响应时间过长,并对查询视频长度敏感.针对上述两个问题,提出了一种基于时空灰度序特征的视频片段定位算法,其关键步骤包括：(1) 在精确定位之前,通过引入线性时间复杂度的基于时空二值模式直方图特征(spatio-temporal binary pattern histogram,简称STBPH)的实时过滤算法以及基于二值时间灰度序特征(binarytemporal ordinal measure,简称BTOM)的快速过滤算法,大幅度减少精确定位阶段需要进行比较的候选视频片段个数;(2) 在精确定位阶段,通过引入唯一性表示能力更好且保持了较好鲁棒性的时空统一灰度序特征(jointspatio-temporal ordinal measure,简称JSTOM)进行序列匹配,显著提高了定位检索的精度.实验结果表明,该算法能够快速、准确地进行视频片段定位,大幅降低了对查询视频长度的敏感度.     

在基于H.323的视频会议中实现负载均衡

基于H.323协议族的纯软件视频会议系统由于其便捷以及成本低廉的优点而得以快速发展。但是当连接用户过多时,服务器就成为系统性能的瓶颈。文中以困扰纯软件视频会议系统的最大问题,服务器负载能力不足为研究方向,提出了在视频会议系统通过转发服务器请求信息通过轮询机制实现负载均衡的方法,并通过静态随机轮询和动态反馈的算法加以实现,从而得到一种提高服务器负载能力的途径。     

Brahma: Browsing and Retrieval Architecture for Hierarchical Multimedia Annotation

Traditional browsing of large multimedia documents (e.g., video, audio) is primarily sequential. In the absence of an index structure browsing and searching for relevant information in a long video, audio or other multimedia document becomes difficult. Manual annotation can be used to mark various segments of such documents. Different segments can be combined to create new annotated segments, thus creating hierarchical annotation structures. Given the lack of structure in media data, it is natural for different users to have different views on the same media data. Therefore, different users can create different annotation structures. Users may also share some or all of each other's annotation structures. The annotation structure can be browsed or used to playback as a composed video consisting of different segments. Finally, the annotation structures can be manipulated dynamically by different users to alter views on a document. BRAHMA is a multimedia environment for browsing and retrieval of multimedia documents based on such hierarchical annotation structures.     

An annotation management system for relational databases

We present an annotation management system for relational databases. In this system, every piece of data in a relation is assumed to have zero or more annotations associated with it and annotations are propagated along, from the source to the output, as data is being transformed through a query. Such an annotation management system could be used for understanding the provenance (aka lineage) of data, who has seen or edited a piece of data or the quality of data, which are useful functionalities for applications that deal with integration of scientific and biological data. We present an extension, pSQL, of a fragment of SQL that has three different types of annotation propagation schemes, each useful for different purposes. The default scheme propagates annotations according to where data is copied from. The default-all scheme propagates annotations according to where data is copied from among all equivalent formulations of a given query. The custom scheme allows a user to specify how annotations should propagate. We present a storage scheme for the annotations and describe algorithms for translating a pSQL query under each propagation scheme into one or more SQL queries that would correctly retrieve the relevant annotations according to the specified propagation scheme. For the default-all scheme, we also show how we generate finitely many queries that can simulate the annotation propagation behavior of the set of all equivalent queries, which is possibly infinite. The algorithms are implemented and the feasibility of the system is demonstrated by a set of experiments that we have conducted.     

Hierarchical Indexing Structure for Efficient Similarity Search in Video Retrieval

With the rapid increase in both centralized video archives and distributed WWW video resources, content-based video retrieval is gaining its importance. To support such applications efficiently, content-based video indexing must be addressed. Typically, each video is represented by a sequence of frames. Due to the high dimensionality of frame representation and the large number of frames, video indexing introduces an additional degree of complexity. In this paper, we address the problem of content-based video indexing and propose an efficient solution, called the ordered VA-file (OVA-file) based on the VA-file. OVA-file is a hierarchical structure and has two novel features: 1) partitioning the whole file into slices such that only a small number of slices are accessed and checked during k nearest neighbor (kNN) search and 2) efficient handling of insertions of new vectors into the OVA-file, such that the average distance between the new vectors and those approximations near that position is minimized. To facilitate a search, we present an efficient approximate kNN algorithm named ordered VA-LOW (OVA-LOW) based on the proposed OVA-file. OVA-LOW first chooses possible OVA-slices by ranking the distances between their corresponding centers and the query vector, and then visits all approximations in the selected OVA-slices to work out approximate kNN. The number of possible OVA-slices is controlled by a user-defined parameter delta. By adjusting delta, OVA-LOW provides a trade-off between the query cost and the result quality. Query by video clip consisting of multiple frames is also discussed. Extensive experimental studies using real video data sets were conducted and the results showed that our methods can yield a significant speed-up over an existing VA-file-based method and (distance with high query result quality. Furthermore, by incorporating temporal correlation of video content, our methods achieved much more efficient performance     

一种基于图模型的Web数据库采样方法

Web数据库中,海量的信息隐藏在具有特定查询能力的查询接口后面,使人无法了解一个Web数据库内容的特征,比如主题的分布、更新的频率等,这就为DeepWeb数据集成带来了巨大的挑战.为了解决这个问题,提出了一种基于图模型的Web数据库采样方法,可以通过查询接口从Web数据库中以增量的方式获取近似随机的样本,即每次查询获取一定数量的样本记录,并且利用已经保存在本地的样本记录生成下一次的查询.该方法的一个重要特点是不受查询接口中属性表现形式的局限,因此是一种一般的Web数据库采样方法.在本地的模拟实验和真实Web数据库上的大量实验表明,该方法可以在较小代价下获得高质量的样本.     

Semantic context based refinement for news video annotation

Automatic video annotation is to bridge the semantic gap and facilitate concept based video retrieval by detecting high level concepts from video data. Recently, utilizing context information has emerged as an important direction in such domain. In this paper, we present a novel video annotation refinement approach by utilizing extrinsic semantic context extracted from video subtitles and intrinsic context among candidate annotation concepts. The extrinsic semantic context is formed by identifying a set of key terms from video subtitles. The semantic similarity between those key terms and the candidate annotation concepts is then exploited to refine initial annotation results, while most existing approaches utilize textual information heuristically. Similarity measurements including Google distance and WordNet distance have been investigated for such a refinement purpose, which is different with approaches deriving semantic relationship among concepts from given training datasets. Visualness is also utilized to discriminate individual terms for further refinement. In addition, Random Walk with Restarts (RWR) technique is employed to perform final refinement of the annotation results by exploring the inter-relationship among annotation concepts. Comprehensive experiments on TRECVID 2005 dataset have been conducted to demonstrate the effectiveness of the proposed annotation approach and to investigate the impact of various factors.