Video frame rate up conversion under inconsistent camera motion

In this paper, we address the problem of video frame rate up-conversion (FRC) in the compressed domain. FRC is often recognized as video temporal interpolation. This problem is very challenging when targeted for video sequences with inconsistent camera and object motion, such as sports videos. A novel compressed domain motion compensation scheme is presented and applied in this paper, aiming at up-sampling frame rates in sports videos. MPEG-2 encoded motion vectors (MVs) are utilized as inputs in the proposed algorithm. The decoded MVs undergo a cumulative spatiotemporal interpolation. An iterative rejection scheme based on the dense motion vector field (MVF) and the generalized affine motion model is exploited to detect global camera motion. Subsequently, the foreground object separation is performed by additionally examining the temporal consistency of the output of iterative rejections. This consistency check process helps coalesce the resulting foreground blocks and weed out the unqualified blocks. Finally, different compensation strategies for the camera and object motions are applied to interpolate the new frames. Illustrative examples are provided to demonstrate the efficacy of the proposed approach. Experimental results are compared with the popular block and non-block based frame interpolation approaches.

Distributed video coding based on part intracoding and soft side information estimation

Recently, distributed source coding (DSC) has been proposed to implement source compression by exploiting source statistics at the decoder only, which enables low-complexity video coding. However, to date, the video codecs based on DSC have been unable to compress as efficiently as traditional predictive video codecs, such as H.264. So, new techniques have to be investigated to improve the performance of the distributed video coding scheme for practical applications. In this paper, I propose a novel distributed video coding scheme based on part intracoding and soft side information estimation. Firstly, at the encoder side, to improve the compression performance of distributed video coding system, we divide the video data into strongly correlative data encoded by Slepian–Wolf codec and weakly correlative data encoded by Intracoding codec. Secondly, at the decoder side, to improve the accuracy of side information estimation, a soft side information estimation method is proposed, which is more suitable for video coding due to the non-stationary feature of video data. Our experimental results show that the performance of our coding system is better than that of the traditional distributed video coding system while keeping the simple encoding property. Also the concept of soft side information is a new idea in distributed video coding and will significantly influence the side information estimation method.

A scalable signature scheme for video authentication

This paper addresses the problem of ensuring the integrity of a digital video and presents a scalable signature scheme for video authentication based on cryptographic secret sharing. The proposed method detects spatial cropping and temporal jittering in a video, yet is robust against frame dropping in the streaming video scenario. In our scheme, the authentication signature is compact and independent of the size of the video. Given a video, we identify the key frames based on differential energy between the frames. Considering video frames as shares, we compute the corresponding secret at three hierarchical levels. The master secret is used as digital signature to authenticate the video. The proposed signature scheme is scalable to three hierarchical levels of signature computation based on the needs of different scenarios. We provide extensive experimental results to show the utility of our technique in three different scenarios—streaming video, video identification and face tampering.

An algorithm to estimate mean vehicle speed from MPEG Skycam video

With low computation cost, motion vectors can be readily extracted from MPEG video streams and processed to estimate vehicle motion speed. A statistical model is proposed to model vehicle speed and noise. In order to achieve high estimation accuracy and also study the limitations of the proposed algorithm, we quantitatively evaluated four parameters used in our algorithm: temporal filter window size T, video resolution R _v (CIF/QCIF), motion vector frame distance m, and video bit-rates. Our experiments showed that the mean vehicle speed can be estimated with high accuracy, up to 85 to 92% by proper spatial and temporal processing. The proposed algorithm is especially suitable for Skycam-based application, where the traditional tracking-based or virtual-loop-based approaches perform poorly because of their requirements of high-resolution images. Although extensive work has been done in extracting motion information directly from MPEG video data in compressed domain, to our best knowledge, this paper is the very first work in which stationary motion (speed) of moving objects can be estimated with high accuracy directly from MPEG motion vectors. Furthermore the proposed method is not limited to vehicle speed estimation by nature and it can be applied to other applications where the stationary motion assumption is satisfied.

The design of a wireless real-time visual surveillance system

In this paper, we study the important issues in the design of an efficient wireless real-time visual surveillance system (WISES). Two important considerations are to minimize: (1) the video workload on the wireless network; and (2) the processing workload at the front-end video capturing unit. To achieve the first objective, we propose a cooperative framework for semantic filtering of video frames instead of forwarding every video frame to the back-end server for analysis and monitoring query evaluation. To minimize the processing workload at the front-end unit, a hierarchical object model (HOM) is designed to model the status of the objects, and their temporal and spatial properties in the video scene. With the information provided from the back-end server, the front-end unit pre-analyses the current status of the objects in the HOM by comparing the selection conditions in the submitted monitoring queries following the adaptive object-based evaluation (APOBE) scheme which is proposed to reduce the processing workload at the front-end unit. In APOBE, a higher evaluation frequency is given to the object which is closer to satisfy the condition in the monitoring queries. The performance of WISES has been studied to demonstrate the efficiency of the proposed scheme.

Video protection for MPEG-4 FGS with watermarking

The MPEG-4 fine granularity scalability (FGS) video coding standard offers flexible adaptation to varying network bandwidths and different application needs. This paper presents a MPEG-4 FGS video CODEC based watermarking scheme to embed watermark during encoding. Watermark is embedded into base layer, and can be extracted from both base layer and enhanced video through eliminating the influence of enhancement layer on watermark. This scheme eliminates error propagation caused by watermark for normal video, and utilizes error propagation caused by watermark adjustment to protect the video content. This scheme provides dual protection for intellectual property rights (IPR): watermark and video content protection utilizing error propagation in temporal motion compensation prediction. Watermark is embedded into I-VOP, and is adjusted before I-VOP is reconstructed as reference VOP. Only customers with authorization can adjust the watermark correctly during decoding to get good video quality. Illegal customers can also access the video, but with bad quality. This scheme has the virtue of providing dual protection with a little expense. Theoretical bounds of watermark embedding strength to keep watermark invisibility and of watermark adjustment strength to get enough protective effect are calculated. Some experimental results are given and analyzed.

Reconfigurable architectures and processors for real-time video motion estimation

With the recent proliferation of multimedia applications, several fast block matching motion estimation algorithms have been proposed in order to minimize the processing time in video coding. While some of these algorithms adopt pre-defined search patterns that directly reflect the most probable motion structures, other data-adaptive approaches dynamically configure the search pattern to avoid unnecessary computations and memory accesses. Either of these approaches leads to rather difficult hardware implementations, due to their configurability and adaptive nature. As a consequence, two different but quite configurable architectures are proposed in this paper. While the first architecture reflects an innovative mechanism to implement motion estimation processors that support fast but regular search algorithms, the second architecture makes use of an application specific instruction set processor (ASIP) platform, capable of implementing most data-adaptive algorithms that have been proposed in the last few years. Despite their different natures, these two architectures provide highly configurable hardware platforms for real-time motion estimation. By considering a wide set of fast and adaptive algorithms, the efficiency of these two architectures was compared and several motion estimators were synthesized in a Virtex-II Pro XC2VP30 FPGA from Xilinx, integrated within a ML310 development platform. Experimental results show that the proposed architectures can be easily reconfigured in run-time to implement a wide set of real-time motion estimation algorithms.

MPEG-4 to H.264 transcoding with frame rate reduction

In this paper, a temporal resolution reduction transcoding method that transforms an MPEG-4 video bitstream into an H.264 video bitstream is proposed. The block modes and motion vectors in the MPEG-4 bitstream are utilized in the H.264 encoder for the block mode conversion and motion vector interpolation methods. Four types of motion vector interpolation methods are proposed in order to avoid the use of brute-force motion estimation in H.264. According to the experimental results, the proposed methods achieve a 3∼4 times improvement in the computational complexity compared to the cascade pixel-domain transcoding method, while the PSNR (peak signal to noise ratio) is degraded by 0.2∼0.9 dB depending on the bitrates.

Motion mapping and mode decision for MPEG-2 to H.264/AVC transcoding

This paper describes novel transcoding techniques aimed for low-complexity MPEG-2 to H.264/AVC transcoding. An important application for this type of conversion is efficient storage of broadcast video in consumer devices. The architecture for such a system is presented, which includes novel motion mapping and mode decision algorithms. For the motion mapping, two algorithms are presented. Both efficiently map incoming MPEG-2 motion vectors to outgoing H.264/AVC motion vectors regardless of the block sizes that the motion vectors correspond to. In addition, the algorithm maps motion vectors to different reference pictures, which is useful for picture type conversion and prediction from multiple reference pictures. We also propose an efficient rate-distortion optimised macroblock coding mode decision algorithm, which first evaluates candidate modes based on a simple cost function so that a reduced set of candidate modes is formed, then based on this reduced set, we evaluate the more complex Lagrangian cost calculation to determine the coding mode. Extensive simulation results show that our proposed transcoder incorporating the proposed algorithms achieves very good rate-distortion performance with low complexity. Compared with the cascaded decoder-encoder solution, the coding efficiency is maintained while the complexity is significantly reduced.

Motion estimation based color transfer and its application to color video compression

In this paper a novel scheme for color video compression using color transfer technique is proposed. Towards this, a new color transfer mechanism for video using motion estimation is presented. Encoder and decoder architectures for the proposed compression scheme are also presented. In this scheme, compression is achieved by firstly discarding chrominance information for all but selected reference frames and then using motion prediction and discrete cosine transform (DCT) based quantization. At decompression stage, the luminance-only frames are colored using chrominance information from the reference frames applying the proposed color transfer technique. To integrate color transfer mechanism with hybrid compression scheme a new color transfer protocol is defined. Both compression scheme and color transfer work in YCbCr color space.

Efficient video encryption scheme based on advanced video coding

A video encryption scheme combining with advanced video coding (AVC) is presented and analyzed in this paper, which is different from the ones used in MPEG1/2 video encryption. In the proposed scheme, the intra-prediction mode and motion vector difference are encrypted with the length-kept encryption algorithm (LKE) in order to keep the format compliance, and the residue data of the macroblocks are encrypted with the residue data encryption algorithm (RDE) in order to keep low cost. Additionally, a key distribution scheme is proposed to keep the robustness to transmission errors, which assigns sub-keys to different frames or slices independently. The encryption scheme’s security, time efficiency and error robustness are analyzed in detail. Experimental results show that the encryption scheme keeps file format unchanged, is secure against replacement attacks, is efficient in computing, and is robust to some transmission errors. These properties make it a suitable choice for real-time applications, such as secure IPTV, secure videoconference or mobile/wireless multimedia, etc.

Real-time 2D to 3D video conversion

We present a real-time implementation of 2D to 3D video conversion using compressed video. In our method, compressed 2D video is analyzed by extracting motion vectors. Using the motion vector maps, depth maps are built for each frame and the frames are segmented to provide object-wise depth ordering. These data are then used to synthesize stereo pairs. 3D video synthesized in this fashion can be viewed using any stereoscopic display. In our implementation, anaglyph projection was selected as the 3D visualization method, because it is mostly suited to standard displays.

Distributed Video Coding using Turbo Trellis Coded Modulation

Distributed Video Coding (DVC) has been proposed for increasingly new application domains. This rise is apparently motivated by the very attractive features of its flexibility for building very low cost video encoders and the very high built-in error resilience when applied over noisy communication channels. Yet, the compression efficiency of DVC is notably lagging behind the state-of-the-art in video coding and compression, H.264/AVC in particular. In this context, a novel coding solution for DVC is presented in this paper, which promises to improve its rate-distortion (RD) performance towards the state-of-the-art. Here, Turbo Trellis Coded Modulation (TTCM), with its attractive coding gain in channel coding, is utilized and its resultant impact in both pixel domain and transform domain DVC framework is discussed herein. Simulations have shown a significant gain in the RD performance when compared with the state-of-the-art Turbo coding based DVC implementations.

Fast adaptive termination mode selection for H.264 scalable video coding

Although scalable video coding can achieve coding efficiencies comparable with single layer video coding, its computational complexity is higher due to its additional inter-layer prediction process. This paper presents a fast adaptive termination algorithm for mode selection to increase its computation speed while attempting to maintain its coding efficiency. The developed algorithm consists of the following three main steps which are applied not only to the enhancement layer but also to the base layer: a prediction step based on neighboring macroblocks, a first round check step, and a second round check step or refinement if failure occurs during the first round check. Comparison results with the existing algorithms are provided. The results obtained on various video sequences show that the introduced algorithm achieves about one-third reduction in the computation speed while generating more or less the same video quality.

Systems, control models, and codec for collaborative observation of remote environments with an autonomous networked robotic camera

Networked robotic cameras are becoming popular in remote observation applications such as natural observation, surveillance, and distance learning. Equipped with a high optical zoom lens and agile pan-tilt mechanisms, a networked robotic camera can cover a large region with various resolutions. The optimal selection of camera control parameters for competing observation requests and the on-demand delivery of video content for various spatiotemporal queries are two challenges in the design of such autonomous systems. For camera control, we introduce memoryless and temporal frame selection models that effectively enable collaborative control of the camera based on the competing inputs from in-situ sensors and users. For content delivery, we design a patch-based motion panorama representation and coding/decoding algorithms (codec) to allow efficient storage and computation. We present system architecture, frame selection models, user interface, and codec algorithms. We have implemented the system and extensively tested our design in real world applications including natural observation, public surveillance, distance learning, and building construction monitoring. Experiment results show that our frame selection models are robust and effective and our on-demand content delivery codec can satisfy a variety of spatiotemporal queries efficiently in terms of computation time communications bandwidth.

An improved block size selection method based on macroblock movement characteristic

The H.264 video compression standard supports seven variable block sizes ranging from 4 × 4 to 16 × 16 for one Macro Block (MB) with 16 × 16 size to conduct motion estimation (ME) and compensation. This new feature achieves significant coding gain at the cost of huge computation complexity. Dozens of fast mode decision algorithms with fast block size selection have been proposed to reduce complexity. In this paper, we propose an improved fast block size selection method based on MB movement characteristic. The Motion Vector (MV) and block residual are employed to analyze the movement characteristic of one MB novelly. Then the movement characteristic is used to decide whether and how to merge or split the MB for encoding. Experimental results show that this method speeds up mode decision procedure dramatically with negligible compression performance degradation.

Transmission of layered video streaming via multi-path on ad hoc networks

Mobile ad hoc networks without centralized infrastructure change their topology rapidly because of node mobility, making multimedia applications difficult to run across wireless networks. Moreover, video transmission over ad hoc networks causes frequent transmission loss of video packets owing to end-to-end transmission with a number of wireless links, and requires essential bandwidth and restricted delay to provide quality-guaranteed display. This paper presents an architecture supporting transmission of multiple video streams in ad hoc networks by establishing multiple routing paths to provide extra video coding and transport schemes. This study also proposes an on-demand multicast routing protocol to transport layered video streams. The multicast routing protocol transmits layered video streaming based on a weight criterion, which is derived according to the number of receivers, delay and expiration time of a route. A simulation is performed herein to indicate the viability and performance of the proposed approach. The simulation results demonstrate that the proposed transport scheme is more effective than other video transport schemes with single or multiple paths.

Video scene segmentation and semantic representation using a novel scheme

Grouping video content into semantic segments and classifying semantic scenes into different types are the crucial processes to content-based video organization, management and retrieval. In this paper, a novel approach to automatically segment scenes and semantically represent scenes is proposed. Firstly, video shots are detected using a rough-to-fine algorithm. Secondly, key-frames within each shot are selected adaptively with hybrid features, and redundant key-frames are removed by template matching. Thirdly, spatio-temporal coherent shots are clustered into the same scene based on the temporal constraint of video content and visual similarity between shot activities. Finally, under the full analysis of typical characters on continuously recorded videos, scene content is semantically represented to satisfy human demand on video retrieval. The proposed algorithm has been performed on various genres of films and TV program. Promising experimental results show that the proposed method makes sense to efficient retrieval of interesting video content.

Real-time stabilization of long range observation system turbulent video

The paper presents a real-time algorithm that compensates image distortions due to atmospheric turbulence in video sequences, while keeping the real moving objects in the video unharmed. The algorithm involves (1) generation of a “reference” frame, (2) estimation, for each incoming video frame, of a local image displacement map with respect to the reference frame, (3) segmentation of the displacement map into two classes: stationary and moving objects; (4) turbulence compensation of stationary objects. Experiments with both simulated and real-life sequences have shown that the restored videos, generated in real-time using standard computer hardware, exhibit excellent stability for stationary objects while retaining real motion.