Fast MPEG-4 Motion Estimation: Processor Based and Flexible VLSI Implementations

MPEG-4 is a new multimedia standard combining interactivity, object-based natural and synthetic digital video, audio and computer-graphics. For the implementation of the video part of the MPEG-4 standard a high degree of flexibility is required, where the motion estimation requires the highest part of the computational power. Therefore, in this paper fast algorithms for MPEG-4 motion estimation are evaluated in terms of visual quality and computational power requirements for processor based implementations. Due to the object-based nature of MPEG-4 also new VLSI architectures for MPEG-4 motion estimation are required. Therefore known motion estimation architectures are evaluated on their capability of being modified for MPEG-4 support. Based on this evaluation a new dedicated, but flexible MPEG-4 motion estimation architecture targeted for low-power handheld applications is presented, which resulted to be advantageous to processor based implementations by magnitudes of order.     

MPEG digital video-coding standards

The efficient digital representation of image and video signals has been the subject of considerable research over the past 20 years. Digital video-coding technology has developed into a mature field and products have been developed that are targeted for a wide range of emerging applications, such as video on demand, digital TV/HDTV broadcasting, and multimedia image/video database services. With the increased commercial interest in video communications, the need for international image- and video-compression standards arose. To meet this need, the Moving Picture Experts Group (MPEG) was formed to develop coding standards. MPEG-1 and MPEG-2 video-coding standards have attracted much attention worldwide, with an increasing number of very large scale integration (VLSI) and software implementations of these standards becoming commercially available. MPEG-4, the most recent MPEG standard that is still under development, is targeted for future content-based multimedia applications. We provide an overview of the MPEG video-coding algorithms and standards and their role in video communications. We review the basic concepts and techniques that are relevant in the context of the MPEG video-compression standards and outline MPEG-1 and MPEG-2 video-coding algorithms. The specific properties of the standards related to their applications are presented, and the basic elements of the forthcoming MPEG-4 standard are also described. We also discuss the performance of the standards and their success in the market place     

Architecture and design of NX-2700: a programmable single-chip HDTVall-format-decode-and-display processor

This paper describes the architecture, functionality, and design of NX-2700, a digital television and media processor chip from Philips Semiconductors. The NX-2700 is the second generation of an architectural family of programmable multimedia processors targeted at the digital television (DTV) markets, including the United States Advanced Television Systems Committee (ATSC) DTV-standard-based applications. The chip not only supports all of the 18 ATSC formats from standard-definition to wide-angle, high-definition video, but has also the power to handle high-definition television (HDTV) video and audio source decoding (high-level MPEG-5 AC-3 and ProLogic audio, closed captioning, etc.) as well as the flexibility to process advanced interactive services. NX-2700 is a programmable processor with a very powerful, general-purpose very long instruction word (VLIW) central processing unit (CPU) core that implements many nontrivial multimedia algorithms, coordinates all on-chip activities, and runs a small real-time operating system. The CPU core, aided by an array of peripheral devices (multimedia coprocessors and input-output units) and high-performance buses, facilitates concurrent processing of audio, video, graphics, and communication-data     

Multimedia processors

This paper describes large-scale-integration programmable processors designed for multimedia processing such as real-time compression and decompression of audio and video as well as the generation of computer graphics. As the target of these processors is to handle audio and video in real time, the processing capability must be increased tenfold compared to that of conventional microprocessors, which were designed to handle mainly texts, figures, tables, and photographs. To clarify the advantages of a high-speed multimedia processing capability, we define these chips as multimedia processors. General-purpose microprocessors for workstations and personal computers (PCs) use special built-in hardware for multimedia processing, so the multimedia processors described include these modified general-purpose microprocessors. After reviewing the history of programmable processors, we classify multimedia processors into five categories depending on their basic architecture. The categories are reduced instruction set computer (RISC) microprocessors for workstations, complex instruction set computer microprocessors for PCs, embedded RISCs, low-power digital signal processors (DSPs), which are mainly used for mobile communications devices, and media processors that support PCs for multimedia applications. These five classes are then grouped into two: microprocessors with a multimedia instruction set and highly parallel DSPs. An architectural comparison between these two groups on the basis of Moving Picture Experts Group decoding applications is made, and the advantages and disadvantages of each class are clarified. Future processors, including “system on a chip,” and their applications are also discussed     

MPEG-4视频验证模型中基于内容的编解码算法

ＭＰＥＧ－４是一个正在制定的编码标准,除了具有 ＭＰＥＧ－１和 ＭＰＥＧ－２标准的基于“帧”的功能以外,ＭＰＥＧ－４视频编码算法还支持多媒体环境中对视频景物内的“物体”进行存取与操纵。文中描述了第四版ＭＰＥＧ－４视频验证模型的结构及其提供的主要编码工具和算法。     

VLSI architectures for video compression-a survey

The paper presents an overview on architectures for VLSI implementations of video compression schemes as specified by standardization committees of the ITU and ISO. VLSI implementation strategies are discussed and split into function specific and programmable architectures. As examples for the function oriented approach, alternative architectures for DCT and block matching will be evaluated. Also dedicated decoder chips are included Programmable video signal processors are classified and specified as homogeneous and heterogenous processor architectures. Architectures are presented for reported design examples from the literature. Heterogenous processors outperform homogeneous processors because of adaptation to the requirements of special, subtasks by dedicated modules. The majority of heterogenous processors incorporate dedicated modules for high performance subtasks of high regularity as DCT and block matching. By normalization to a fictive 1.0 μm CMOS process typical linear relationships between silicon area and through-put rate have been determined for the different architectural styles. This relationship indicates a figure of merit for silicon efficiency     

Architectural enhancements for network congestion control applications

Complex network protocols and various network services require significant processing capability for modern network applications. One of the important features in modern networks is differentiated service. Along with differentiated service, rapidly changing network environments result in congestion problems. In this paper, we analyze the characteristics of representative congestion control applications-scheduling and queue management algorithms, and we propose application-specific acceleration techniques that use instruction-level parallelism (ILP) and packet-level parallelism (PLP) in these applications. From the PLP perspective, we propose a hardware acceleration model based on detailed analysis of congestion control applications. In order to get large throughputs, a large number of processing elements (PEs) and a parallel comparator are designed. Such hardware accelerators provide large parallelism proportional to the number of processing elements added. A 32-PE enhancement yields 24/spl times/ speedup for weighted fair queueing (WFQ) and 27/spl times/ speedup for random early detection (RED). For ILP, new instruction set extensions for fast conditional operations are applied for congestion control applications. Based on our experiments, proposed architectural extensions show 10%-12% improvement in performance for instruction set enhancements. As the performance of general-purpose processors rapidly increases, defining architectural extensions (e.g., multi-media extensions (MMX) as in multimedia applications) for general-purpose processors could be an alternative solution for a wide range of network applications.     

HiBRID-SoC: A Multi-Core SoC Architecture for Multimedia Signal Processing

The HiBRID-SoC multi-core system-on-chip architecture targets a wide range of multimedia applications with particularly high processing demands, including general signal processing applications, video de-/encoding, image processing, or a combination of these tasks. For this purpose, the HiBRID-SoC integrates three fully programmable processors cores and various interfaces onto a single chip, all tied to a 64-Bit AMBA AHB bus. The processor cores are individually optimized to the particular computational characteristics of different application fields, complementing each other to deliver high performance levels with high flexibility at reduced system cost. The HiBRID-SoC is fabricated in a 0.18 μm 6LM standard-cell CMOS technology, occupies about 81 mm², and operates at 145 MHz. An MPEG-4 Advanced Simple Profile decoder in full D1 resolution requires about 120 MHz for real-time operation on the HiBRID-SoC, utilizing only two of the three cores. Together with the third core, a custom region-of-interest (ROI) based surveillance application can be built.Hans-Joachim Stolberg received the Dipl.-Ing. degree in electrical engineering from the University of Hannover, Germany, in 1995.From 1995 to 1996, he was with the NEC Information Technology Research Laboratories, Kawasaki, Japan, working on efficient implementations of video compression algorithms. Since 1996, he has been with the Institute of Microelectronic Systems at the University of Hannover as a Research Assistant. During summer 2001, he was a Monbukagakusho Research Fellow at the Tokyo Institute of Technology, Japan. His current research interests include VLSI architectures for video signal processing, performance estimation of multimedia schemes, and profile-guided memory organization for signal processing and multimedia applications.Mladen Bereković received the Dipl.-Ing. degree in electrical engineering from the University of Hannover, Germany, in 1995.Since then he has been a Research Assistant with the Institute of Microelectronic Systems of the University of Hannover. His current research interests include VLSI architectures for video signal processing, MPEG-4, System-on-Chip (SOC) designs, and simultaneously multi-threaded (SMT) processor architectures.Sören Moch received the Dipl.-Ing. degree in electrical engineering from the University of Hannover, Germany, in 1997.Since then he has been Research Assistant with the Laboratory for Information Technology, University of Hannover. His current research interests are in the area of processor architectures for image, video and multimedia signal processing applications.Lars Friebe studied electrical engineering at the Universities Ulm and Hannover, Germany. In 1999, he worked at the NEC System ULSI Research Laboratory in Kanagawa, Japan. He received the Dipl.-Ing. degree in electrical engineering from the University of Hannover, Germany, in 1999.Since then he has been a Research Assistant with the Laboratory for Information Technology, University of Hannover. His current research interests are in the area of parallel programmable VLSI architectures for real-time image processing.Mark B. Kulaczewski started his studies in electrical engineering at the University of Hannover, Germany. In 1994, he transferred to Purdue University, West Lafayette, USA, and received the M.S. degree in electrical engineering in 1996.Since 1997 he has been a Research Assistant at the Laboratory for Information Technology and the Institute of Microelectronic Systems, University of Hannover. His current research interests include programmable real-time architectures for video coding and image segmentation, and instruction-set extensions for cryptographic applications.Sebastian Flügel was born in Crivitz, Germany, in 1975. He received his Dipl.-Ing. degree from the Department of Electrical Engineering of the University of Rostock in 2001.Since then he has been a Ph.D. candidate at the Institute of Microelectronic Systems at the University of Hannover. He works in the field of architectures and systems for video processing systems. His focus is on algorithms for video encoding and the development of optimized hardware architectures.Heiko Klußmann received the Dipl.-Ing. degree in computer engineering from the University of Hannover, Germany, in 2002.Since then he has been a Research Assistant with the Institute of Microelectronic Systems of the University of Hannover. His current research interests are in the area of programmable architectures for real-time video signal processing.Andreas Dehnhardt was born in Frankfurt am Main, Germany, in 1976. He received his Dipl.-Ing. degree in electrical engineering from the University of Hannover, Germany, in 2002.Since then, he has been a Research Assistant with the Institute of Microelectronic Systems, University of Hannover. His current research interests include programmable architectures for multimedia applications and implementation of real-time MPEG-4 encoding schemes.Peter Pirsch received the Ing. grad. degree from the engineering college in Hannover, Hannover, Germany, in 1966, and the Dipl.-Ing. and Dr.-Ing. degrees from the University of Hannover, in 1973 and 1979, respectively, all in electrical engineering.From 1966 to 1973 he was employed by Telefunken, Hannover, working in the Television Department. He became a Research Assistant at the Department of Electrical Engineering, University of Hannover, in 1973, a Senior Engineer in 1978. During 1979 to 1980 and in Summer 1981 he was on leave, working in the Visual Communications Research Department, Bell Laboratories, Holmdel, NJ. During 1983 to 1986 he was Department Head for Digital Signal Processing at the SEL research center, Stuttgart. Since 1987 he is Professor in the Department of Electrical Engineering, since 2002 in the Department of Computer Science at the University of Hannover. He served as Vice President Research of the University of Hannover from 1998 to 2002. His present research includes architectures and VLSI implementations for image processing applications, rapid prototyping and design automation for DSP applications. He is the author or coauthor of more than 200 technical papers. He has edited a book on VLSI Implementations for Image Communications (Elsevier 1993) and is author of the book Architectures for Digital Signal Processing (John Wiley 1998).Pirsch is a member of the IEEE, the German Institute of Information Technology Engineers (ITG) and the German Association of Engineers (VDI). He was recipient of several awards: the NTG paper price award (1982), IEEE Fellow (1997), IEEE Circuits and Systems Golden Jubilee Medal (1999). He was member or chair of several technical program committees of international conferences and organizer of special sessions and preconference courses. He has held several administrative and technical positions with the IEEE Circuits and Systems Society and other professional organizations. Dr. Pirsch currently serves as Vice President Publications of the IEEE Circuits and Systems Society. Since 2000 he is chairman of the Accreditation Commission for Engineering and Informatics of the Accreditation Agency for Study Programs in Engineering, Informatics, Natural Science and Mathematics (ASIIN). Dr. Pirsch is chair of the VDI committee on Engineering Education.     

Progressive Watermarking Techniques Using Genetic Algorithms

An innovative watermarking scheme based on progressive transmission with genetic algorithms (GAs) is proposed. We implement the watermarking embedding and extraction systems in the transform domain, and apply the JPEG spectral selection mode for scalable transmission of the watermarked image. By employing a GA with a proper fitness function into the watermarking system, both the watermark imperceptibility and watermark robustness requirements are considered and optimized. The number of embedded bits, or the watermark capacity, is much larger than that in other existing algorithms in the literature. Also, the watermark capacity in the proposed algorithm lies within the theoretical limit. In addition, the embedded watermark can be partly extracted at the receiver side even when the watermarked image is being transmitted. Simulation results show both the robustness and the effectiveness of progressive transmission under different attacking schemes and different bandwidth variations. The proposed scheme is directly applicable to scalable coding of multimedia, such as MPEG-4 scalable video coding.     

Multimedia Signal Processors: An Architectural Platform with Algorithmic Compilation

Novel algorithmic features of multimedia applications and advances in VLSI technologies are driving forces behind the new multimedia signal processors. We propose an architecture platform which could provide high performance and flexibility, and would require less external I/O and memory access. It is comprised of array processors to be used as the hardware accelerator and RISC cores to be used as the basis of the programmable processor. It is a hierarchical and scalable architecture style which facilitates the hardware-software codesign of multimedia signal processing circuits and systems. While some control-intensive functions can be implemented using programmable CPUs, other computation-intensive functions can rely on hardware accelerators.To compile multimedia algorithms, we also present an operation placement and scheduling scheme suitable for the proposed architectural platform. Our scheme addresses data reusability and exploits local communication in order to avoid the memory/communication bandwidth bottleneck, which leads to faster program execution. Our method shows a promising performance: a linear speed-up of 16 times can be achieved for the block-matching motion estimation algorithm and the true motion tracking algorithm, which have formed many multimedia applications (e.g., MPEG-2 and MPEG-4).     

MPEG-4 natural audio coding

MPEG-4 audio represents a new kind of audio coding standard. Unlike its predecessors, MPEG-1 and MPEG-2 high-quality audio coding, and unlike the speech coding standards which have been completed by the ITU-T, it describes not a single or small set of highly efficient compression schemes but a complete toolbox to do everything from low bit-rate speech coding to high-quality audio coding or music synthesis. The natural coding part within MPEG-4 audio describes traditional type speech and high-quality audio coding algorithms and their combination to enable new functionalities like scalability (hierarchical coding) across the boundaries of coding algorithms. This paper gives an overview of the basic algorithms and how they can be combined.     

Parallel Algorithms Based on the Temporal-Window Method for Non-Alternating 3D-WT over Angiographies Using a Multicomputer

In this paper, we introduce and evaluate the parallel implementations of two video sequences decorrelation algorithms having been developed based on the non-alternating three-dimensional wavelet transform (3D-WT) and the temporal-window method. The proposed algorithms have been proven to outperform the classic 3D-WT algorithm in terms of a better coding efficiency and lower computational requirements while enabling a lossless coding and a top-quality reconstruction: the two most highly relevant features to medical imaging applications. The parallel implementations of the algorithms are developed and tested on a shared memory system, a SGI Origin 3800 supercomputer, making use of a message-passing paradigm. We evaluate and analyze the performance of the implementations in terms of the response time and speed-up factor by varying the number of processors and various video coding parameters. The key point enabling the development of highly efficient implementations rely on a workload distribution strategy supplemented by the use of parallel I/O primitives, for better exploiting the inherent features of the application and computing platform. Two sets of I/O primitives are tested and evaluated: the ones provided by the C compiler and the ones belonging to the MPI/IO library.     

MPEG-4 video verification model: A video encoding/decoding algorithm based on content representation

MPEG-4 video aims at providing standardized core technologies allowing efficient storage, transmission and manipulation of video data in multimedia environments. This is a challenging task given the broad spectrum of requirements and applications in multimedia. In order to achieve this broad goal, rather than a solution for a narrow set of applications, functionalities common to clusters of applications are under consideration. Therefore, video group activities in MPEG-4 aim at providing solutions in the form of tools and algorithms enabling functionalities such as efficient compression, object scalability, spatial and temporal scalability, and error resilience. The standardized MPEG-4 video will provide a toolbox containing tools and algorithms bringing solutions to the above-mentioned functionalities and more.

The current focus of the MPEG-4 video group is the development of the Video Verification Models. A Verification Model (VM) is a common platform with a precise definition of encoding and decoding algorithms which can be presented as tools addressing specific functionalities. It evolves through time by means of core experiments. New algorithms/tools are added to the VM and old algorithms/tools are replaced in the VM by successful core experiments. Until October 1996, the MPEG-4 video group has focused its efforts on a single VM which has gradually evolved from version 1.0 to version 4.0, and in the process has addressed increasing number of desired functionalities, namely, content based object and temporal scalabilities, spatial scalability, error resilience, and compression efficiency.

This paper gives an overview of version 4.0 of the Video VM in MPEG-4. In doing so, issues, algorithms, and majors tools used in the development of this future video standard are discussed.     

MPEG-4编解码算法及其视频性能分析

3G移动终端要求能够提供包含语音、数据和图像的多媒体业务，为此3GPP制定了第一个适合3G系统的基于电路交换的多媒体终端模型——3G-324M规范，该模型支持MPEG-4视频处理，它具有高压缩比和容错编码特性。在研究MPEG-4编解码算法基础上进行了MPEG-4视频编码的性能测试。     

MPEG-4视频对象分割技术

随着MPEG-4,MPEG-7的研究发展,其基于内容的编码和面向对象的存取和操纵技术日益得到人们的重视。基于对象的视频图像分割是实现MPEG-4基于内容的编码和交互功能的关键。视频图像分割方法分为自动分割法和半自动分割法两种。结合视频分割的发展趋势,深入介绍了基于对象的视频分割的主要技术及国内外的最新研究算法,包括数学形态学算法以及活动轮廓模型(蛇模型)在该领域的应用,并分析了当前视频分割技术尚存在的问题和研究前景。     

An Algorithm-Hardware-System Approach to VLIW Multimedia Processors

Very Long Instruction Word (VLIW) processor architectures for multimedia applications are discussed from an algorithm, hardware and system based point of view. VLIW processors show high flexibility and processing power, as well as a good utilization of resources by compiler-generated code, but their exclusive exploitation of instruction level parallelism (ILP) decreases in efficiency as the degree of parallelism increases. This is mainly caused by characteristics of multimedia algorithms, increasing wiring delays, compiler restrictions, and a widening gap between on-chip processing speed and available bandwidth to external memory. As new multimedia applications and standards continue to evolve (MPEG-4), the demand for higher processing power will continue. Therefore, parallel processing in all its available forms will have to be exploited to achieve significant performance improvements. We show that, due to the diminishing returns from a further increase in ILP, multimedia applications will benefit more from an additional exploitation of parallelism at thread-level. We examine how simultaneous multithreading (SMT), a novel architectural approach combining VLIW techniques with parallel processing of threads, can efficiently be used to further increase performance of typical multimedia workloads.     

基于对象的视频编码——MPEG－4

MPEG－4是一种新的基于对象的视频编码标准,可以广泛应用于多媒体通信中。文章从MPEG－4的视频压缩原理出发,介绍MPEG－4中采用的主要压缩方法及抗误码技术。     

Methodologies used for evaluation of video tools and algorithms in MPEG-4

MPEG-4 issued two calls for proposals requesting submission of algorithms and tools relevant to standardization of MPEG-4. This paper reports on the evaluation of tools submitted for evaluation in November 1995 and January 1996. Complete video coding schemes submitted in January 1996 are also covered. The goal of the evaluation was to cluster the tools according to the technical areas they address, to evaluate them according to the issues relevant to the standardization process, and finally to suggest areas of core experiments to improve a video verification model (VM) as soon as the VM becomes available. Altogether, MPEG evaluated 87 tools and 19 complete coding algorithms, most of them highlighted in this paper. During the evaluation, 19 areas for core experiments were identified. Each core experiment is targeted at different functionalities like compression efficiency, content-based coding, error resilience, scalability. This definition of core experiments caused close collaboration and supported mutual fertilization between organizations working on similar tools, which allowed the VM to progress much faster than expected.     

Processing the new world of interactive media

Trimedia is a family of programmable multimedia processors from the Trimedia product group of Philips Semiconductors. This architecture is based upon a high-performances VLIW CPU core. TM-1000 is the first product from a family of multimedia processors based upon the Trimedia architecture. TM-1000 is designed to concurrently process video, audio, graphics, and communication data. TM-1000 consists of a high-performance VLIW-based CPU core, large instruction and data caches, main memory interface, and video, audio and communication related peripherals. TM-1000 is a multimedia system on a chip: high-quality video and audio applications can be implemented in TM-1000 using high-level languages such as “C” and “C++”. The authors mainly focus on the VLIW CPU architecture     

System-Level Energy-Delay Exploration for Multimedia Applications on Embedded Cores with Hardware Cache

Program transformations are a powerful way of optimizing given applications for lower power and higher performance. In this paper, we explore avenues for power reduction by program transformations using the real-time constraints. In the sequel, we discuss the effects of our methodology, for optimization of power, on cache related performance aspects. Our target applications are in the real-time multimedia applications domain implemented on programmable multimedia or DSP processors. The effectiveness of our approach in obtaining a low power implementation and real-time performance is illustrated on three real-life applications, viz. a MPEG-2 decoder, a QSDPCM video codec and a Voicecoder application. Our experimental results indeed show that we are able to obtain lower power and still achieve a real-time performance.