ViewStation applications: implications for network traffic

This paper describes applications built on the ViewStation, a distributed multimedia system based on Unix workstations and a gigabit per second local area network. A key tenet of the ViewStation project is the delivery of media data not just to the desktop but all the way to the application program. As processing power continues to improve, our approach enables applications that perform intensive processing of audio and video data. We hypothesize that as media data are shaped by this software-based processing, the resultant network traffic patterns will be dominated more by software behavior than by so-called real-time issues. We have written applications that directly process live video to provide more responsive human-computer interaction. We have also developed applications to explore the potential of media processing to support content-based retrieval of prerecorded television broadcasts. These applications perform intelligent processing on video, as well as straightforward presentation. They demonstrate the utility of network-based multimedia systems that deliver audio and video data all the way to the application. The network requirements of the applications are modeled as a combination of bursty transfers and periodic packet-trains     

视频分割算法的研究

视频对象分割是视频处理中的难点问题,它在基于内容的检索、对象识别和交互操作的多媒体中有重要应用。视频对象分割技术具有重要的研究意义和应用价值。本文主要分析了2种视频分割算法,包括基于时空联合的视频分割算法、基于运动一致性的视频分割算法。基于时空联合的视频分割算法是充分利用时间和空间信息,以获得满意的分割效果;基于运动一致性的视频分割算法是选用稳健的多分辨率平均位移帧差平方最小方法来估计视频对象的参数化运动。     

Next-generation content representation, creation, and searching fornew-media applications in education

Content creation, editing, and searching are extremely time-consuming tasks that often require substantial training and experience, especially when high-quality audio and video are involved. New media represents a new paradigm for multimedia information representation and processing, in which the emphasis is placed on the actual content. It thus brings the tasks of content creation and searching much closer to actual users and enables them to be active producers of audio-visual information rather than passive recipients. We discuss the state of the art and present next-generation techniques for content representation, searching, creation and editing. We discuss our experiences in developing a Web-based distributed compressed video editing and searching system (WebClip), a media-representation language (Flavor) and an object-based video authoring system (Zest) based on it, and a large image/video search engine for the World Wide Web (WebSEEk). We also present a case study of new media applications based on specific planned multimedia education experiments with the above systems in several K-12 schools in Manhattan, NY     

Multimedia friendly home wireless LANs

Home wireless local area networks (LANs) will carry diverse multimedia applications such as data, video, voice, and time sensitive control information. To enable quality delivery of such applications, these networks should incorporate an efficient quality of service (QoS) support mechanism. However, existing home wireless LANs do not provide support for multimedia applications. In this paper, we introduce our software based solution that provides QoS support for voice, video and data while using existing wireless home networks. The experimental results we have obtained provide evidence that our solution provides QoS support for multimedia applications.     

Semi-automatic knowledge extraction, representation and context-sensitive intelligent retrieval of video content using collateral context modelling with scalable ontological networks

Automatic indexing and retrieval of digital data poses major challenges. The main problem arises from the ever increasing mass of digital media and the lack of efficient methods for indexing and retrieval of such data based on the semantic content rather than keywords. To enable intelligent web interactions, or even web filtering, we need to be capable of interpreting the information base in an intelligent manner. For a number of years research has been ongoing in the field of ontological engineering with the aim of using ontologies to add such (meta) knowledge to information. In this paper, we describe the architecture of a system (Dynamic REtrieval Analysis and semantic metadata Management (DREAM)) designed to automatically and intelligently index huge repositories of special effects video clips, based on their semantic content, using a network of scalable ontologies to enable intelligent retrieval. The DREAM Demonstrator has been evaluated as deployed in the film post-production phase to support the process of storage, indexing and retrieval of large data sets of special effects video clips as an exemplar application domain. This paper provides its performance and usability results and highlights the scope for future enhancements of the DREAM architecture which has proven successful in its first and possibly most challenging proving ground, namely film production, where it is already in routine use within our test bed Partners’ creative processes.     

Automatic video segmentation and tracking for content-based applications

Advanced multimedia applications have to provide content-related functionalities such as search and retrieval of meaningful objects, detection and analysis of events, and understanding of scenes, which allow the user to access and manipulate the multimedia content with greater flexibility. This greatly depends on automatic techniques for extracting such objects from multimedia data. In this article we intend to provide a tutorial on the state-of-the-art in video segmentation and tracking technology with particular attention paid to the recent developments in attention-based object extraction. Performance results are included to highlight this emerging technology     

A joint signal processing and cryptographic approach to multimedia encryption.

In recent years, there has been an increasing trend for multimedia applications to use delegate service providers for content distribution, archiving, search, and retrieval. These delegate services have brought new challenges to the protection of multimedia content confidentiality. This paper discusses the importance and feasibility of applying a joint signal processing and cryptographic approach to multimedia encryption, in order to address the access control issues unique to multimedia applications. We propose two atomic encryption operations that can preserve standard compliance and are friendly to delegate processing. Quantitative analysis for these operations is presented to demonstrate that a good tradeoff can be made between security and bitrate overhead. In assisting the design and evaluation of media security systems, we also propose a set of multimedia-oriented security scores to quantify the security against approximation attacks and to complement the existing notion of generic data security. Using video as an example, we present a systematic study on how to strategically integrate different atomic operations to build a video encryption system. The resulting system can provide superior performance over both generic encryption and its simple adaptation to video in terms of a joint consideration of security, bitrate overhead, and friendliness to delegate processing.     

A Scheme for Intelligent Image Retrieval in Multimedia Databases

Storage and retrieval of visual data play an important role in multimedia systems. We have developed a content-based scheme for retrieving images from multimedia databases intelligently. The retrieval takes two stages. The first stage retrieves an image based on partial information. In the second stage, the system accumulates knowledge from the results of the first-stage retrieval. It analyzes the subspace of features from the resulting images and tries to understand the query request. It also makes full use of the entire index space, although queries can be made on partial information. The technology developed will find many applications in multimedia areas. It will also provide a tool for studying how humans rank the similarity of images and what information people use in visual perception, etc., and will help in the development of methods based on these human approaches.     

基于嵌入式零树小波编码直方图图像检索

图像和视频应用的快速增长，使得根据图像和视频内容进行查询的技术变得越来越重要，人们提出了许多基于像素域或压缩域的图像检索技术，因为多媒体数据库通常具有相当大的数据量，所以基于像素域图像检索技术的计算复杂度相当大，因此，许多文献提出更快的基于压缩域的图像检索技术，本文提出一种改进的基于嵌入式零树小波编码直方图的图像检索技术，特征提取综合考虑图像的颜色，纹理，频率和空间信息，所有的特征可以在压缩过程中自动得到，图像检索的过程就是匹配待检索图像和来自数据库的侯选图像的索引，实验证明这种方法具有好的检索性能。     

基于异构多流网络的多模态人体动作识别

人体动作识别在人机交互、视频内容检索等领域有众多应用,是多媒体信息处理的重要研究方向。现有的大多数基于双流网络进行动作识别的方法都是在双流上使用相同的卷积网络去处理RGB与光流数据,缺乏对多模态信息的利用,容易造成网络冗余和相似性动作误判问题。近年来,深度视频也越来越多地用于动作识别,但是大多数方法只关注了深度视频中动作的空间信息,没有利用时间信息。为了解决这些问题,本文提出一种基于异构多流网络的多模态动作识别方法。该方法首先从深度视频中获取动作的时间特征表示,即深度光流数据,然后选择合适的异构网络来进行动作的时空特征提取与分类,最后对RGB数据、RGB中提取的光流、深度视频和深度光流识别结果进行多模态融合。通过在国际通用的大型动作识别数据集NTU RGB+D上进行的实验表明,所提方法的识别性能要优于现有较先进方法的性能。      

Applications of entropic spanning graphs

This article presents applications of entropic spanning graphs to imaging and feature clustering applications. Entropic spanning graphs span a set of feature vectors in such a way that the normalized spanning length of the graph converges to the entropy of the feature distribution as the number of random feature vectors increases. This property makes these graphs naturally suited to applications where entropy and information divergence are used as discriminants: texture classification, feature clustering, image indexing, and image registration. Among other areas, these problems arise in geographical information systems, digital libraries, medical information processing, video indexing, multisensor fusion, and content-based retrieval.     

DHSR: A Novel Semantic Retrieval Approach for Ubiquitous Multimedia

Semantic features are critical intelligence information for mobile ubiquitous multimedia, how to manage and retrieve the semantic information has been an important issue. In this paper, a novel semantic retrieval approach named Data Hiding based Semantic Retrieval (DHSR) for ubiquitous multimedia is proposed. This approach consists of the following features: (1) Every multimedia document has to be semantically annotated by several users before saved into multimedia database. (2) Semantic information described by object ontology will be hidden in the multimedia document data. (3) Semantic information will not be lost even if the multimedia document is copied, cut or leave the database. Our work provides a search engine with convenient user interfaces. The experimental results show that DHSR can search the multimedia documents reflecting users’ query intent more effectively compared with some traditional approaches.     

Generic content-based audio indexing and retrieval framework

Rapid increase in the amount of the digital audio collections presenting various formats, types, durations and other parameters that the digital multimedia world refers demands a generic framework for robust and efficient indexing and retrieval based on the aural content. Moreover, from the content-based multimedia retrieval point of view, the audio information can be even more important than the visual part as it is mostly unique and significantly stable within the entire duration of the content. A generic and robust audio-based multimedia indexing and retrieval framework, which has been developed and tested under the MUVIS system, is presented. This framework supports the dynamic integration of the audio feature extraction modules during the indexing and retrieval phases and therefore provides a test-bed platform for developing robust and efficient aural feature extraction techniques. Furthermore, the proposed framework is designed based on the high-level content classification and segmentation in order to improve the speed and accuracy of the aural retrievals. Both theoretical and experimental results are finally presented, including the comparative measures of retrieval performance with respect to the visual counterpart.     

图像搜索引擎检索模式分析与研究

文中研究如何从HTML文档中提取图片相关信息,保证高效和准确的实现图片检索。在对图像搜索引擎检索模式分析的基础上,提出了若干关键技术,设计并实现了一个基于文本的Web图片搜索引擎,给出了系统的总体结构图．并对获取网页、提取信息、图片抓取、建立索引和提供查询进行了详细的描述,分析了图像搜索引擎的检索模式。     

MPEG-4 and rate-distortion-based shape-coding techniques

We address the problem of the efficient encoding of object boundaries. This problem is becoming increasingly important in applications such as content-based storage and retrieval, studio and television postproduction, and mobile multimedia applications. The MPEG-4 visual standard will allow the transmission of arbitrarily shaped video objects. The techniques developed for shape coding within the MPEG-4 standardization effort are described and compared first. A framework for the representation of shapes using their contours is presented next. Such representations are achieved using curves of various orders, and they are optimal in the rate-distortion sense. Finally, conclusions are drawn