Hidden Markov Model Inversion for Audio-to-Visual Conversion in an MPEG-4 Facial Animation System

MPEG-4 standard allows composition of natural or synthetic video with facial animation. Based on this standard, an animated face can be inserted into natural or synthetic video to create new virtual working environments such as virtual meetings or virtual collaborative environments. For these applications, audio-to-visual conversion techniques can be used to generate a talking face that is synchronized with the voice. In this paper, we address audio-to-visual conversion problems by introducing a novel Hidden Markov Model Inversion (HMMI) method. In training audio-visual HMMs, the model parameters {_av} can be chosen to optimize some criterion such as maximum likelihood. In inversion of audio-visual HMMs, visual parameters that optimize some criterion can be found based on given speech and model parameters {_av}. By using the proposed HMMI technique, an animated talking face can be synchronized with audio and can be driven realistically. The HMMI technique combined with MPEG-4 standard to create a virtual conference system, named VIRTUAL-FACE, is introduced to show the role of HMMI for applications of MPEG-4 facial animation.     

MPEG-4: Why,what, how and when?

The MPEG-4 Version 1 standard has been recently finalized. Since MPEG-4 adopted an object-based audiovisual representation model with hyperlinking and interaction capabilities and supports both natural and synthetic content, it is expected that this standard will become the information coding playground for future multimedia applications. This paper intends to give an overview on the MPEG-4 motivations, objectives, achievements, process and workplan, providing a stimulating starting point for more detailed reading.     

MPEG-4 multimedia for our time

The MPEG-4 standard explores every possibility of the digital environment. Recorded images and sounds co-exist with their computer-generated counterparts, a new language for sound promises compact-disk quality at extremely low data rates; and the multimedia content could even adjust itself to suit the transmission rate and quality. Possibly the greatest of the advances made by MPEG-4 is that viewers and listeners need no longer be passive. The height of “interactivity” in audiovisual systems today is the users ability merely to stop or start a video in progress. MPEG-4 is completely different: it allows the user to interact with objects within the scene, whether they derive from so-called real sources, such as moving video, or from synthetic sources, such as computer-aided design output or computer-generated cartoons. Authors of content can give users the power to modify scenes by deleting, adding, or repositioning objects, or to alter the behavior of the objects. Perhaps the most immediate need for MPEG-4 is defensive. It supplies tools with which to create uniform (and top-quality) audio and video encoders and decoders on the Internet, preempting what may become an unmanageable tangle of proprietary formats. In addition to the Internet, the standard is also designed for low bit-rate communications devices, which are usually wireless     

Insights into low-level avatar animation and MPEG-4 standardization

Referring to the new functionality of video access and coding, the survey presented here lies within the scope of MPEG-4 activities related to virtual character (VC) animation. We first describe how Amendment 1 of the MPEG-4 standard offers an appropriate framework for virtual human animation, gesture synthesis and compression/transmission. Specifically, face and body representation and animation are described in detail in terms of node syntax and animation stream encoding methods. Then, we discuss how this framework is extended within the ongoing standardization efforts by (1) allowing the animation of any kind of articulated model, and (2) addressing advanced modeling and animation concepts as “skin and bones”-based approach. The new syntax for node definition and animation stream is presented and discussed in terms of genericity and additional functionalities. The biomechanical properties, modeled by means of the character skeleton that defines the bone influence on the skin region, as well as the local spatial deformations simulating muscles, are supported by specific nodes. Animating the VC consists in instantiating bone transformations and muscle control curve. Interpolation techniques, inverse kinematics, discrete cosine transform and arithmetic encoding techniques make it possible to provide a highly compressed animation stream. The new animation framework extension tools are finally evaluated in terms of realism, complexity and transmission bandwidth within a sign language communication system.     

ATSC Video and Audio Coding

In recent decades, digital video and audio coding technologies have helped revolutionize the ways we create, deliver, and consume audiovisual content. This is exemplified by digital television (DTV), which is emerging as a captivating new program and data broadcasting service. This paper provides an overview of the video and audio coding subsystems of the Advanced Television Systems Committee (ATSC) DTV standard. We first review the motivation for data compression in digital broadcasting. The MPEG-2 video and AC-3 audio compression algorithms are described, with emphasis on basic concepts, system features, and coding performance. Next-generation video and audio codecs currently under consideration for advanced services are also presented.     

The MPEG-4 Multimedia Coding Standard: Algorithms, Architectures and Applications

The upcoming MPEG-4 standard provides new possibilities for the compression and presentation of multimedia contents. The main characteristics of MPEG-4 are the object-based coding and representation of an audio-visual scene and the ability to code objects of natural or synthetic origin. These features will enhance existing applications with new functionalities and enable standardised solutions for new applications. This paper provides an overview of the three major parts Systems, Visual and Audio of the new MPEG-4 standard, highlights implementation aspects for some envisaged types of MPEG-4 terminals and describes possible future multimedia application scenarios using MPEG-4 functionalities.     

Two-dimensional mesh-based visual-object representation forinteractive synthetic/natural digital video

This paper first provides an overview of two-dimensional (2-D) and three-dimensional mesh models for digital video processing. It then introduces 2-D mesh-based modeling of video objects as a compact representation of motion and shape for interactive, synthetic/natural video manipulation, compression, and indexing. The 2-D mesh representation and the mesh geometry and motion compression have been included in the visual tools of the upcoming MPEG-4 standard. Functionalities enabled by 2-D mesh-based visual-object representation include animation of still texture maps, transfiguration of video overlays, video morphing, and shape-and motion-based retrieval of video objects     

网络化多媒体实时监控系统的应用研究

采用MPEG－1编码解码国际标准和TCP／IP网络协议,在网络环境下,实现视音频信息和控制数据的网上传输。介绍了网络化多媒体实时监控系统的的构成、工作原理和功能特点,并对视音频实时编码技术和网络编程技术进行了详细探讨。     

AVS中的音视频编码压缩技术

介绍了音视频编码标准AVS中的主要技术特点,对AVS标准所采用的主要技术进行了综述,给出了AVS视频标准与MPEG-4 AVC/H.264编码器性能的比较和分析,讨论了AVS的发展前景.     

MPEG-4 and rate-distortion-based shape-coding techniques

We address the problem of the efficient encoding of object boundaries. This problem is becoming increasingly important in applications such as content-based storage and retrieval, studio and television postproduction, and mobile multimedia applications. The MPEG-4 visual standard will allow the transmission of arbitrarily shaped video objects. The techniques developed for shape coding within the MPEG-4 standardization effort are described and compared first. A framework for the representation of shapes using their contours is presented next. Such representations are achieved using curves of various orders, and they are optimal in the rate-distortion sense. Finally, conclusions are drawn     

Fast MPEG-4 Motion Estimation: Processor Based and Flexible VLSI Implementations

MPEG-4 is a new multimedia standard combining interactivity, object-based natural and synthetic digital video, audio and computer-graphics. For the implementation of the video part of the MPEG-4 standard a high degree of flexibility is required, where the motion estimation requires the highest part of the computational power. Therefore, in this paper fast algorithms for MPEG-4 motion estimation are evaluated in terms of visual quality and computational power requirements for processor based implementations. Due to the object-based nature of MPEG-4 also new VLSI architectures for MPEG-4 motion estimation are required. Therefore known motion estimation architectures are evaluated on their capability of being modified for MPEG-4 support. Based on this evaluation a new dedicated, but flexible MPEG-4 motion estimation architecture targeted for low-power handheld applications is presented, which resulted to be advantageous to processor based implementations by magnitudes of order.     

MPEG-4: a multimedia standard for the third millennium. 1

MPEG-4 (formally ISO/IEC international standard 14496) defines a multimedia system for the interoperable communication of complex scenes containing audio, video, synthetic audio and graphics material. In this article, we provide a comprehensive overview of the technical elements of the Moving Pictures Expert Group's MPEG-4 multimedia system specification     

MPEG-4标准起草工作近况

本文简略报道国际活动图像专家组近年对多媒体通信草拟新标准MPEG 4的进展情况。其中音频编码包括语音、音乐 (自然的和合成的 ) ,比特率从 2至 6 4kb /s。视频编码包括甚低比特率 5～ 6 4kb /s和较高比特率 6 4kb /s至 2Mb /s。视频编码可将图像中每一对象分开编码为不同的比特流层 ,又可操纵对象的尺度、位置等 ,具有以内容为基础的交互式功能。除了核心编码器外 ,对输入视频序列的每一帧分成若干个任意形状的“视频对象平面” ,编码成各个分开的“视频对象层”。另外 ,利用“子图形”编码技术 ,将图像的背景以及前景中的每一对象分开编码成视频序列后传输 ,可以改善视像质量。     

Instruction Set Extensions for MPEG-4 Video

This paper describes instruction set extensions for the acceleration of MPEG-4 algorithms on programmable (RISC-) CPUs. MPEG-4 standardizes audio and video compression schemes for a variety of bit rates and scenarios. As MPEG-4 targets a much broader range of different applications than previously defined hybrid video coding standards like H.263 or MPEG-2, it employs a much higher number of different algorithms and coding modes. Therefore, MPEG-4 implementations will require a more software-oriented approach to be efficient. However, the total computational load for an optimized implementation of an MPEG-4 video codec is expected to exceed the performance levels of today's multimedia signal processors, making further hardware acceleration a necessity. For that purpose, we propose a number of instruction set extensions that add function-specific blocks to the data path of a CPU. These dedicated modules are highly adapted to the most computation-intensive processing schemes of MPEG-4, such as DCT, motion compensation, padding, shape coding, or bitstream parsing. The increased functionality of basic instructions results in a significant speed-up over standard RISC instruction sets, thus making MPEG-4 implementations feasible on programmable processor platforms. Possible target architectures include VLIW multimedia processors, MIMD-style multiprocessors, or coprocessor architectures     

MPEG-4 natural audio coding

MPEG-4 audio represents a new kind of audio coding standard. Unlike its predecessors, MPEG-1 and MPEG-2 high-quality audio coding, and unlike the speech coding standards which have been completed by the ITU-T, it describes not a single or small set of highly efficient compression schemes but a complete toolbox to do everything from low bit-rate speech coding to high-quality audio coding or music synthesis. The natural coding part within MPEG-4 audio describes traditional type speech and high-quality audio coding algorithms and their combination to enable new functionalities like scalability (hierarchical coding) across the boundaries of coding algorithms. This paper gives an overview of the basic algorithms and how they can be combined.     

MPEG-4视频验证模型中基于内容的编解码算法

ＭＰＥＧ－４是一个正在制定的编码标准,除了具有 ＭＰＥＧ－１和 ＭＰＥＧ－２标准的基于“帧”的功能以外,ＭＰＥＧ－４视频编码算法还支持多媒体环境中对视频景物内的“物体”进行存取与操纵。文中描述了第四版ＭＰＥＧ－４视频验证模型的结构及其提供的主要编码工具和算法。     

MPEG: a technological basis for multimedia applications

The Moving Picture Experts Group was established in 1988 in the framework of the Joint ISO/IEC Technical Committee (JTC 1) on Information Technology. They were given the mandate to develop standards for coded representation of moving pictures, associated audio, and their combination when used for storage and retrieval on digital storage media (DSM). In April 1990 they became Working Group 11 of JTC 1/SC 2, then in November 1991, WG 11 of JTC 1/SC 29. The three original work items of the group-coding up to 1.5, 10, and 40 Mbps-were nicknamed MPEG-1, -2 and -3, respectively. The group dropped the MPEG3 work item in July 1992 when it became apparent that the functionalities supported by the MPEG-2 requirements made it redundant. The current MPEG-4 work item, first proposed in May 1991 and approved in July 1993, targets audiovisual coding at very low bit-rates