Efficient motion vector prediction algorithm using pattern matching

The state-of-the-art median prediction scheme is widely used for predicting motion vectors (MVs) in recent video standards. By exploiting the spatial correlations among MVs, median prediction scheme predicts MV for current block from three neighboring blocks. When MV is obtained from motion estimation, MV difference (MVD) is calculated and then transmitted. This process for predicting MV and calculating MVD is known as MV coding process. For MV coding, the performance depends on how efficient both the spatial and the temporal correlations among MVs are being exploited. Median prediction scheme applies a sophisticated way including some special rules to exploit the spatial correlations, however the temporal correlations among successive MVs are not exploited. In this paper, a new algorithm named MV pattern matching (MV-PM) exploiting both the spatial and temporal correlations is proposed. Various kinds of experimental results show that the proposed MV-PM algorithm outperforms the median prediction and the other related prediction schemes.     

Iterative search strategy with selective bi-directional prediction for low complexity multiview video coding

The multiview video coding (MVC) extension of H.264/AVC is the emerging standard for compression of impressive 3D and free-viewpoint video. The coding structure in MVC adopts motion and disparity estimation to exploit temporal and inter-view dependencies in MVC. It results in a considerable increase in encoding complexity. Most of the computational burden comes from uni-directional and bi-directional prediction. In this paper, an iterative search strategy is designed to speed up the uni-directional prediction in MVC. It can work with an adaptive search range adjustment through a confidence measure of a loop constraint to obtain both motion and disparity vectors jointly. Furthermore, a selective bi-directional prediction algorithm is proposed to enhance the coding performance by analyzing the statistical characteristics of bi-directional prediction in MVC. Experimental results demonstrate that, by using the proposed fast search, the temporal and inter-view redundancies of multiview video can be eliminated sufficiently with low complexity.     

一种新的视点间预测结构

多视点视频编码除应具有较高的编码效率外,还应该包括后向兼容性、时间随机访问和视点可分级性等,这些都主要取决于所采用的预测结构。目前所提供的多视点视频编码(Joint Multi-view Video Coding, JMVC)采用固定的视点间预测结构,难以适应复杂情况的多视点视频编码。该文综合考虑编码效率和用户随机访问等因素,根据多视点视频相关性分析自适应调整视点间预测结构,以获得较好的编码综合性能。试验结果表明,与JMVC相比,该文的方法在提高编码效率的同时,有较好的随机访问性能。     

Efficient early direct mode decision for multi-view video coding

Multi-view video coding (MVC) uses various prediction modes and exhaustive mode decision to achieve high coding efficiency. However, the introduced heavy computational complexity becomes the bottleneck of the practical application of MVC. For this, an efficient early Direct mode decision for MVC is proposed in this paper. Based on the observation that the Direct mode is highly possible to be the optimal mode, the proposed method first computes the rate distortion (RD) cost of the Direct mode and compares this RD cost value with an adaptive threshold for providing an early termination chance as follows. If this RD cost value is smaller than the adaptive threshold, the Direct mode will be selected as the optimal mode and the checking process of the remaining modes will be skipped; otherwise, all the modes will be checked to select the one with the minimum RD cost as the optimal mode. Note that the above-mentioned adaptive threshold is determined as the median prediction value of a set of thresholds, which are derived by using the spatial, temporal and inter-view correlations between the current macroblock (MB) and its neighboring MBs, respectively. Experimental results have demonstrated that the proposed method is able to significantly reduce the computational complexity of MVC with negligible loss of coding efficiency, compared with the exhaustive mode decision in MVC.     

Multi-view video coding with view interpolation prediction for 2D camera arrays

An efficient compression algorithm for multi-view video sequences, which are captured by two-dimensional (2D) camera arrays, is proposed in this work. First, we propose a novel prediction structure, called three-dimensional hierarchical B prediction (3DHBP), which can efficiently reduce horizontal inter-view redundancies, vertical inter-view redundancies, and temporal redundancies in multi-view videos. Second, we develop a view interpolation scheme based on the bilateral disparity estimation. The interpolation scheme yields high quality view frames by adapting disparity estimation and compensation procedures using the information in neighboring frames. Simulation results demonstrate that the proposed multi-view video coding algorithm provides significantly better rate–distortion (R–D) performance than the conventional algorithm, by employing the 3DHBP structure and using interpolated view frames as additional reference frames.     

多视点视频编码中图像分组技术的研究

在多视点视频编码中,引入了视点间运动补偿预测技术,而在视点间预测过程中,由于视差矢量的使用,使得GoP长度对编码效率的影响与一般H.264/AVC编码有所不同。在分析了可分级B帧的预测结构对多视点视频编码性能的影响基础上,通过对不同视频序列设置不同GoP的大小,经过多次反复实验,而后对实验结果进行分析,找到了在一定数值范围内的GoP。通过该GoP得到了较好的PSNR和模值较小的MV或者DV,节约了一定的编码时间。     

Fast macroblock encoding algorithm based on rate-distortion activity for multiview video coding

Multiview video coding (MVC) is the appendix H of H.264/AVC, and it requires a great amount of time to compress multiple viewpoints׳ video with complex prediction structures. To reduce the whole computational complexity of MVC, this paper proposes a fast macroblock (MB) encoding algorithm based on rate-distortion (RD) activity, and it includes the fast mode decision and the fast motion/disparity estimation. First, the RD activity type of the current MB is calculated by utilizing the Skip/Direct RD cost and the average RD costs of classified MB modes. Then, through utilizing the RD activity type and RD costs of the estimated modes, the selection of candidate modes, the early decision of Skip/Direct mode, and the reduction of Inter8×8 mode estimation are all presented in the fast mode decision. By using the RD activity type and the correlations of vectors, the selection of search center and the prediction of search range are introduced in the fast motion/disparity estimation. In addition, the proposed algorithm can be applied to temporal and inter-view views as well as anchor and non-anchor frames. An experiment with a wide range of video scenes, camera setups and quantization parameters was implemented, and the results confirmed that the proposed algorithm can reduce the encoding time significantly while maintaining a similar RD performance as the original MVC encoder. Compared to the state-of-the-art algorithms, the proposed algorithm also demonstrated better performances in the various test cases.     

Fast synthesized and predicted just noticeable distortion maps for perceptual multiview video coding

The just noticeable distortion (JND) map is a useful tool for perceptual video coding. However, direct calculation of the JND map incurs high complexity, and the problem is aggravated in multiview video coding. In this paper, two fast methods are proposed to generate the JND maps of multiview videos. In the first method, the JND maps of some anchor views are used to synthesize the JND maps of other views via the depth image based rendering (DIBR), which can be much faster than direct JND computation. In the second method, the motion and disparity vectors obtained during the video coding are employed to predict the JND maps. If the prediction is not satisfactory, the JND block will be refreshed by calculating the JND directly. This method does not need any camera parameters and depth maps. The performances of the two fast JND map generation methods are evaluated in a perceptual MVC framework, where the residuals after spatial, temporal, or inter-view prediction are tuned according to the JND thresholds to save the bits without affecting the perceptual quality. Experimental results show that the JND prediction method has better accuracy and lower complexity. In addition, both fast JND methods lead to negligible degradation of the coding performance, compared to the direct JND method.     

Fast motion and disparity estimation for HEVC based 3D video coding

The emerging international standard for high efficiency video coding (HEVC) based 3D video coding (3D-HEVC) is an extension of HEVC. In the test model of 3D-HEVC, variable size motion estimation (ME) and disparity estimation (DE) are both employed to select the best coding mode for each treeblock in the encoding process. This technique achieves the highest possible coding efficiency, but it brings extremely high computational complexity which limits 3D-HEVC from practical applications. In this paper, a fast ME/DE algorithm based on inter-view and spatial correlations is proposed to reduce 3D-HEVC computational complexity. Since the multi-view videos represent the same scene with similar characteristic, there is a high correlation among the coding information from inter-view prediction. Besides, the homogeneous regions in texture video have a strong spatial correlation, and thus spatially neighboring treeblocks have similar coding information. Therefore, we can determine ME search range and skip some specific ME and DE rarely used in the previously coded view frames and spatially neighboring coding unit. Experimental results demonstrate that the proposed algorithm can significantly reduce computational complexity of 3D-HEVC encoding while maintaining almost the same rate-distortion performance.     

A Regular VLSI Architecture of Motion Vector Prediction for Multiple-Standard MPEG-Like Video Codec

Motion vector (MV) prediction and residue coding technique is adopted to fully utilize the motion field redundancy in the prevailing video standards, and MV prediction is desired in both video encoder and decoder. The computation burden for MV prediction is not very high. However, there is high irregularity in raw MV prediction algorithm with two-stage and four-level hierarchical tree control flows. It makes efficient VLSI architecture implementation challenging. The high irregularity is mainly derived from the abundant inter prediction modes including variable block size partition and temporal prediction direction, as well as the irregular control flow of the MV prediction algorithm. This paper proposes a highly regular architecture to implement MV prediction for multi-standard video codec. Complex control logic is simplified by regularly table look-up of the control parameters predefined and stored in on-chip tables. The parameters of the current macroblock (MB) and its neighboring blocks are initialized and refreshed in a regular manner. Moreover, pipelining and parallelism are employed in the proposed architecture to improve throughput efficiency and tradeoff between hardware cost and efficiency. Simulation results verify the effectiveness of the proposed design.     

Joint depth-motion dense estimation for multiview video coding

The multiview video coding (MVC) extension of H.264/MPEG-4 AVC [1] is one of the most promising visual encoders for three-dimensional television and free viewpoint video applications. In this paper, we propose a joint dense motion/disparity estimation algorithm, designed to replace the classical temporal/inter-view unit within MVC, which uses a block-based motion/disparity estimation. The motion vector fields and the disparity vector fields are therefore simultaneously derived using the stereo-motion consistency constraint in a set theoretic convex optimization framework. The obtained displacement vector fields are then jointly segmented by minimizing a rate-distortion cost function, in line with the multiple reference frame strategy used in H.264/MPEG-4 AVC. Experimental results demonstrate the benefits of the proposed method compared to the separated dense estimation scheme or the block-based estimation technique.     

Selective Disparity Estimation and Variable Size Motion Estimation Based on Motion Homogeneity for Multi-View Coding

Multi-view video coding (MVC) is an ongoing standard in which variable size disparity estimation (DE) and motion estimation (ME) are both employed to select the best coding mode for each macroblock (MB). This technique achieves the highest possible coding efficiency, but it results in extremely large encoding time which obstructs it from practical use. In this paper, a fast DE and ME algorithm based on motion homogeneity is proposed to reduce MVC computational complexity. The basic idea of the method is to utilize the spatial property of motion field in prediction where DE and variable size ME are needed, and only in these regions DE and variable size ME are enabled. The motion field is generated by the corresponding motion vectors (MVs) in spatial window. Simulation results show that the proposed algorithm can save 63% average computational complexity, with negligible loss of coding efficiency.      

A novel temporal error concealment method based on fuzzy reasoning for H.264

In this paper, a fuzzy reasoning based temporal error concealment method is proposed. The basic temporal error concealment is implemented by estimating Motion Vector (MV) of the lost MacroBlock (MB) from its neighboring MVs. Which MV is the most proper one is evaluated by some criteria. Generally, two criteria are widely used, namely Side Match Distortion (SMD) and Sum of Absolute Difference (SAD) of corresponding MV. However, each criterion could only partly describe the status of lost block. To accomplish the judgement more accurately, the two measures are considered together. Thus a refined measure based on fuzzy reasoning is adopted to balance the effects of SMD and SAD. Terms SMD and SAD are regarded as fuzzy input and the term ‘similarity’ as output to complete fuzzy reasoning. Result of fuzzy reasoning repre-sents how the tested MV is similar to the original one. And k-means clustering technique is performed to de-fine the membership function of input fuzzy sets adaptively. According to the experimental results, the con-cealment based on new measure achieves better performance.     

基于多视点视频编码的差错控制算法

 多视点视频编码(Multiview Video Coding,MVC)利用运动估计和视差估计取得了较好的编码性能,但在易错的网络环境下传输MVC视频码流,将导致差错在视点内与视点间进行扩散.针对多视点视频的编码特性,提出了一种端到端的失真度估计模型,并将此模型与率失真优化相结合得到一种基于联合信源信道的编码模式选择算法.实验结果表明该方法能够在易错网络环境下有效的提高多视点视频的传输效率.     

一种低复杂度立体视频错误隐藏新方法

针对立体视频流传输中右视 点整帧丢失,提出 了一种低复杂度的错误隐藏算法。首先,为了高效地感知立体视频的时域质量和视点间质量 ,定义了时域相似尺度(TSM)、 视间相似尺度(ISM)的概念;将前一时刻右视点图像进行时域和视点间匹配,分别求取 其以像素为单位的TSM和 ISM映射图;然后,计算前一时刻右视点图像当前宏块的TSM和ISM值,通过比较得 到当前宏块的预测模式;最后,根据视频序列的时域一致性,将前一时刻右视点图像宏块 的预测模式作为丢失图像宏 块的预测模式,从而使用运动补偿预测(MCP)或者视差补偿预测(DCP )的方法恢复丢失信息。研究结果表明,与已有错误隐藏 算法相比,本文算法获得更好主客观视觉效果;同时与基于图像结构相似度(SSIM)的错误隐藏算法相比,在保持主观视觉质量情况下,错误隐藏时间节省20%左右。     

Adaptive Learning Based View Synthesis Prediction for Multi-View Video Coding

In the applications of Free View TV, pre-estimated depth information is available to synthesize the intermediate views as well as to assist multi-view video coding. Existing view synthesis prediction schemes generate virtual view picture only from interview pictures. However, there are many types of signal mismatches caused by depth errors, camera heterogeneity or illumination difference across views and these mismatches decrease the prediction capability of virtual view picture. In this paper, we propose an adaptive learning based view synthesis prediction algorithm to enhance the prediction capability of virtual view picture. This algorithm integrates least square prediction with backward warping to synthesize the virtual view picture, which not only utilizes the adjacent views information but also the temporal decoded information to adaptively learn the prediction coefficients. Experiments show that the proposed method reduces the bitrates by up to 18 % relative to the multi-view video coding standard, and about 11 % relative to the conventional view synthesis prediction method.     

An efficient rate–distortion optimization method for dependent view in MV-HEVC based on inter-view dependency

Rate–distortion optimization (RDO) is utilized to select the optimal coding parameters in multi-view video coding (MVC), which employs a Lagrange multiplier to balance the relationship between the distortion and the bitrate. In this paper, an efficient RDO method for the dependent view (DV) in multi-view video (MVV) is proposed based on inter-view dependency. First of all, by investigating the sources of the distortion in the DV, a new distortion model for the DV is established. In addition, based on the proposed distortion model, an efficient Lagrangian multiplier decision for B frame is proposed by considering the inter-view dependency. Finally, the optimized Lagrangian multiplier for P frame is designed using the scaling factor which is deduced to have a linear relationship with the disparity between I frame and P frame. Experiment results demonstrate that compared with the original HTM-16.0 encoder, the proposed overall method reduces 12.19% BD-rate for the DV on average, bringing 0.40 dB BD-PSNR gain.     

Error Concealment Using Inter‐layer Correlation for Scalable Video Coding

In this paper, we propose a new error concealment (EC) method using inter‐layer correlation for scalable video coding. In the proposed method, the auxiliary motion vector (MV) and the auxiliary mode number (MN) of intra prediction are interleaved into the bitstream to recover the corrupted frame. In order to reduce the bit rate, the proposed method encodes the difference between the original and the predicted values of the MV and MN instead of the original values. Experimental results show that the proposed EC outperforms the conventional EC by 2.8 dB to 6.7 dB.     

一种低功率分层运动估值器的VLSI结构设计

本文提出一种新的低功率分层运动估值器的VLSI结构,它支持低比特视频编码器的高级预测模式,如H.263和MPEG-4。为减少芯片尺寸及功率消耗,在所有搜索层中使用同一个基本的搜索单元 (BSU)。另外,通过对数据流的有效控制,使其在高级预测模式下,在获得宏块运动矢量的同时,也获得每个宏块中的4个88子块的运动矢量。实验结果表明,这种结构采用较少的门电路,有效降低了功率消耗,并且实现了与全搜索块匹配算法(FSBMA)相似的编码效果,可广泛应用于无线视频通信所需的低功率视频编码器中。     

Cross-layer optimized authentication and error control for wireless 3D medical video streaming over LTE

3D video for tele-medicine applications is gradually gaining momentum since the 3D technology can provide precise location information. However, the weak link for 3D video streaming is the necessary wireless link of the communication system. Neglecting the wireless impairments can severely degrade the performance of 3D video streaming that communicates complex critical medical data. In this paper, we propose systematic methodology for ensuring high performance of the 3D medical video streaming system. First, we present a recursive end-to-end distortion estimation approach for MVC (multiview video coding)-based 3D video streaming over error-prone networks by considering the 3D inter-view prediction. Then, based on the previous model, we develop a cross-layer optimization scheme that considers the LTE wireless physical layer (PHY). In this optimization, the authentication requirements of 3D medical video are also taken into account. The proposed cross-layer optimization approach jointly controls and manages the authentication, video coding quantization of 3D video, and the modulation and channel coding scheme (MCS) of the LTE wireless PHY to minimize the end-to-end video distortion. Experimental results show that the proposed approach can provide superior 3D medical video streaming performance in terms of peak signal-to-noise ratio (PSNR) when compared to state-of-the-art approaches that include joint source-channel optimized streaming with multi-path hash-chaining based-authentication, and also conventional video streaming with single path hash-chaining-based authentication.