Efficient multi-view video coding using inter-view information

Multi-view video coding (MVC) has been extended from H.264/AVC to improve the coding efficiency of multi-view video. This paper proposes a fast mode decision algorithm which can make an early decision on the correct mode partition to solve the issue of the enormous computational complexity. The best modes of the reference views are utilized to determine the complexity of the macroblock (MB) in the current view, the mode candidates needed to be calculated can then be obtained according to the complexity. If the complexity is low or medium, the search range can be reduced. The threshold of the rate-distortion cost for the current MB is calculated using the co-located and neighboring MBs in previously coded view and is utilized as the criterion for early termination. The motion vector difference in the reference view is applied to dynamically adjust the search range in the current MB. Experimental results prove that the proposed algorithm achieves a time saving of 81.05% for a fast TZ search and 87.85% for full search, and still maintains quality performance and bitrate.     

Error concealment analysis for H.264/advanced video coding encoded video sequences

Error concealment methods have become very important in particular when transmitting video streams over error prone wireless links. Often a retransmission of corrupted sequences is not possible and thus the receiver has to make the best out of the received stream. The contributions of this article are the following: firstly, a performance comparison of various error concealment strategies (straight decoding, slice level concealment and macroblock level concealment) is presented based on the detection of errors, the exact location of which is unknown. Secondly, an analytical treatment of the slice level concealment, resulting in a precise mathematical model is provided. Finally, further improvements are proposed by subjective methods based on visual inspection and comparison of their performance by means of simulations.     

Error control and concealment for video communication: a review

The problem of error control and concealment in video communication is becoming increasingly important because of the growing interest in video delivery over unreliable channels such as wireless networks and the Internet. This paper reviews the techniques that have been developed for error control and concealment. These techniques are described in three categories according to the roles that the encoder and decoder play in the underlying approaches. Forward error concealment includes methods that add redundancy at the source end to enhance error resilience of the coded bit streams. Error concealment by postprocessing refers to operations at the decoder to recover the damaged areas based on characteristics of image and video signals. Last, interactive error concealment covers techniques that are dependent on a dialogue between the source and destination. Both current research activities and practice in international standards are covered     

Error concealment algorithms for robust decoding of MPEG compressed video

This paper provides some research results on the topic of error resilience for robust decoding of MPEG (Moving Picture Experts Group) compressed video. It introduces and characterizes the performance of a general class of error concealment algorithms. Such receiver-based error concealment techniques are essential for many practical video transmission scenarios such as terrestrial HDTV broadcasting, packet network based teleconferencing/multimedia, and digital SDTV/HDTV delivery via ATM (asynchronous transfer mode). Error concealment is intended to ameliorate the impact of channel impairments (i.e., bit-errors in noisy channels, or cell-loss in ATM networks) by utilizing available picture redundancy to provide a subjectively acceptable rendition of affected picture regions. The concealment process must be supported by an appropriate transport format which helps to identify image pixel regions which correspond to lost or damaged data. Once the image regions (i.e., macroblocks, slices, etc.) to be concealed are identified, a combination of spatial and temporal replacement techniques may be applied to fill in lost picture elements. A specific class of spatio-temporal error concealment algorithms for MPEG video is described and alternative realizations are compared via detailed end-to-end simulations for both one- or two-tier transmission media. Several algorithm enhancements based on directional interpolation, ‘I-picture motion vectors’, and use of MPEG-2 ‘scalability’ features are also presented. In each case, achievable performance improvements are estimated via simulation. Overall, these results demonstrate that the proposed class of error concealment algorithms provides significant robustness for MPEG video delivery in the presence of channel impairments, permitting useful operation at ATM cell-loss rates in the region of 10⁻⁴ to 10⁻³ and 10⁻² to 10⁻¹ for one- and two-tier transmission scenarios, respectively.     

Error prevention and concealment for scalable video coding with dual-priority transmission

In this work, we present an efficient error resilient system against ATM cell loss using a hybrid error concealment and error propagation prevention (ECP) technique with dual-priority transmission scheme (DPTS). DPTS performs traffic policing to form dual-priority cells in ATM connections and manages to make most cell losses occur in a low priority layer. However, cell loss may still occur in the high priority layer if the bandwidth is not reserved enough for the usually variable bitrate video traffic. Therefore, the ECP technique can still be utilized to reduce the error damage and limit the impact of cell loss to the erroneous slices. Simulation results of two-layer MPEG-2 coding over DPTS in ATM networks demonstrate that ECP with feedback over DPTS can effectively isolate errors and reduce the damage to yield a satisfactory performance, even when the cell-loss rate is as high as 8%.     

Error concealment for shape in MPEG-4 object-based video coding.

In asynchronus transfer mode networks, cell loss or channel errors can cause data to be dropped in the channel. When digital images/videos are transmitted over these networks, one must be able to reconstruct the missing data so that the impact of the errors is minimized. In this paper, we present an error-concealment technique for shape in MPEG-4 object-based video coding. This method, which is based on using global motion estimation and compensation techniques for boundary recovery, consists of three steps: (1) boundary extraction from shape; (2) boundary patching using global motion compensation; and (3) boundary filling to reconstruct the shape of the damaged video object planes. Global motion parameters are inserted as part of the USER_DATA field in the compressed stream and are utilized in reconstructing the damaged boundaries of compressed video object planes.     

Selective recovery of video packet loss using error concealment

An efficient recovery method using error concealment is proposed for video packet loss in fast packet switching networks. In this method, the receiver detects the damaged picture area caused by packet loss from the structured picture data received, makes error concealments, notifies the transmitter, and continues decoding. The transmitter, having received the notice, calculates the affected picture area in the local decoded picture and continues encoding without using this affected area. In this selective recovery method, video signals are not stopped even if a long propagation delay exists, no additional information is transmitted to error recovery and conventional coding algorithms can be used. The proposed method is suitable for multipoint communication. Simulation results show the affected picture area is localized for a considerable time attesting to the method's effectiveness     

Error concealment for video transmission with dual multiscale Markov random field modeling

A novel error concealment algorithm based on a stochastic modeling approach is proposed as a post-processing tool at the decoder side for recovering the lost information incurred during the transmission of encoded digital video bitstreams. In our proposed scheme, both the spatial and the temporal contextual features in video signals are separately modeled using the multiscale Markov random field (MMRF). The lost information is then estimated using maximum a posteriori (MAP) probabilistic approach based on the spatial and temporal MMRF models; hence, a unified MMRF-MAP framework. To preserve the high frequency information (in particular, the edges) of the damaged video frames through iterative optimization, a new adaptive potential function is also introduced in this paper. Comparing to the existing MRF-based schemes and other traditional concealment algorithms, the proposed dual MMRF (DMMRF) modeling method offers significant improvement on both objective peak signal-to-noise ratio (PSNR) measurement and subjective visual quality of restored video sequence.     

多视点视频中基于乘加误差模型的亮度和色差校正

由于相机性能的差异,多视点视频序列之间总是存在亮度和颜色的差异,这种差异会影响多视点视频压缩和显示等后续处理过程的性能.本文由此提出了一种适用于多视点视频的亮度和色差(亮色度)校正算法.根据相机成像原理,不同相机的亮色度差异可以归类为乘性误差和加性误差.灰度级偏移,相机电子元件的放大和量化因子差异等都可以由乘加误差模型表示.为降低计算复杂度,本文采用直方图形状匹配的方法计算乘加误差模型的参数.该算法可作为多视点视频系统中的预处理过程,实验结果表明,该算法能够有效地提高后续压缩过程的性能.     

Error concealment using affine transform for H.263 coded videotransmissions

A new error concealment method is proposed that uses motion estimation to consider actual motions, such as rotation, magnification, reduction, and parallel motion, in moving pictures. Since many videos include a variety of complex three-dimensional motions, the proposed method uses an affine transform to estimate the motion of lost data more accurately, thereby producing a higher peak signal-to-noise ratio value and better subjective video quality     

Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline

Content production for stereoscopic 3D-TV displays has become mature in the past years while huge progress has also been achieved in the improvement of the image quality of glasses-free auto-stereoscopic displays and light-field displays. Concerning the latter two display families, the content production workflow is less elaborated and more complex, as the number of required views not only differs considerably but is also likely to increase in the near future. As a co-existence of all 3D display families can be expected for the next years, one aims to establish an efficient content production workflow which yields to high quality content for all 3D-TV displays.Against this background we present a real-time capable multi-view video plus depth (MVD) content production workflow based on a four-camera rig with mixed narrow and wide baseline. Results show the suitability of the approach to simultaneously produce high quality MVD4 and native stereoscopic 3D content.     

Error concealment techniques for digital TV

Compressed video bitstreams are intended for real-time transmission over communication networks. Most of the video coding standards employ the temporal and spatial prediction structure to reduce the transmitted video data. Therefore, the coded video bitstreams are highly sensitive to information loss and channel errors. Even a single bit error can lead to disastrous quality degradation in both time and space. This quality deterioration is exacerbated when no error resilient coding mechanism is employed to protect coded video data against the error prone environments. Error concealment is a data recovery technique that enables the decoder to conceal effects of transmission errors by predicting the lost or corrupted video data from the previously reconstructed error-free information. Motion vector recovery and motion compensation with the estimated motion vector is a good approach to conceal the corrupted macroblock data. In this paper, we develop various error concealment algorithms based on motion vector recovery, and compare their performances to those of conventional error concealment methods.     

Color calibration of multi-view video plus depth for advanced 3D video

Multi-view video plus depth (MVD) format is considered as the next-generation standard for advanced 3D video systems. MVD consists of multiple color videos with a depth value associated with each texture pixel. Relying on this representation and by using depth-image-based rendering techniques, new viewpoints for multi-view video applications can be generated. However, since MVD is captured from different viewing angles with different cameras, significant illumination and color differences can be observed between views. These color mismatches degrade the performance of view rendering algorithms by introducing visible artifacts leading to a reduced view synthesis quality. To cope with this issue, we propose an effective method for correcting color inconsistencies in MVD. Firstly, to avoid occlusion problems and allow performing correction in the most accurate way, we consider only the overlapping region when calculating the color mapping function. These common regions are determined using a reliable feature matching technique. Also, to maintain the temporal coherence, correction is applied on a temporal sliding window. Experimental results show that the proposed method reduces the color difference between views and improves view rendering process providing high-quality results.     

Error concealment technique based on optical flow

A new motion vector recovery algorithm based on optical flow is proposed. Optical flow fields are very similar to the true motion and can be used to recover three-dimensional motion information. Experimental results on test video sequences produced a higher peak signal-to-noise ratio value than that of conventional methods based on the block matching algorithm     

Efficient view-temporal prediction structures for multi-view video coding

To compress multi-view video, spatial redundancy between adjacent view sequences as well as temporal redundancy need to be eliminated. View-temporal prediction structures are proposed, which can be adjusted to various characteristics of multi-view videos. The proposed prediction structure achieves better coding performance than the reference prediction structure for the standardisation of multi-view video coding.     

Temporal concealment of video transmission errors usinggrid-deformation motion model

A novel temporal concealment technique that uses a grid-deformation motion model is proposed. A triangular mesh is superimposed on the damaged area, grid nodes are displaced according to a temporal prediction, and an affine transform is applied to interpolate the missing area. Experimental results show significant improvements compared to previously proposed approaches     

Spatial shape error concealment for object-based image and video coding

In this paper, an original spatial shape error-concealment technique, to be used in the context of object-based image and video coding schemes, is proposed. In this technique, it is assumed that the shape of the corrupted object at hand is in the form of a binary alpha plane, in which some of the shape data is missing due to channel errors. From this alpha plane, a contour corresponding to the border of the object can be extracted. However, due to errors, some parts of the contour will be missing and, therefore, the contour will be broken. The proposed technique relies on the interpolation of the missing contours with Bézier curves, which is done based on the available surrounding contours. After all the missing parts of the contour have been interpolated, the concealed alpha plane can be easily reconstructed from the fully recovered contour and used instead of the erroneous one improving the final subjective impact.     

Architectures for multi-threaded MVC-compliant multi-view video decoding and benchmark tests

3D video based on stereo/multi-view representations is becoming widely popular. Real-time encoding/decoding of such video is an important concern as the number and spatial/temporal resolution of views increase. We present a systematic method for design and optimization of multi-threaded multi-view video encoding/decoding algorithms using multi-core processors and provide benchmark results for real-time decoding. The proposed multi-core decoding architectures are compliant with the current MVC extension of H.264/AVC international standard, and enable multi-threaded processing with negligible loss of encoding efficiency and minimum processing overhead. Benchmark results show that multi-core processors and multi-threading decoding are necessary for real-time high-definition multi-view video decoding and display.     

基于H．263视频解码的错误检测和掩盖

本文提出了一种应用于H．263视频编码传输的纠错方法,即基于H．263视频解码的错误检测和掩盖算法。这种算法可应用于公共交换电话网（PSTN）,Internet或移动信道的视频通信。模拟结果表明,这种纠错方法在不降低编码效率的前提下,使在有噪声干扰的信道上传输图像的质量有显著的改善。     

A video error concealment method using data hiding based on compressed sensing over lossy channel

The video error concealment with data hiding (VECDH) method aims to conceal video errors due to transmission according to the auxiliary data directly extracted from the received video file. It has the property that can well reduce the error propagated between spatially/temporally correlated macro-blocks. It is required that, the embedded information at the sender side should well capture/reflect the video characteristics. Moreover, the retrieved data should be capable of correcting video errors. The existing VECDH algorithms often embed the required information into the corresponding video frames to gain the transparency. However, at the receiver side, the reconstruction process may loss important information, which could result in a seriously distorted video. To improve the concealment performance, we propose an efficient VECDH algorithm based on compressed sensing (CS) in this paper. For the proposed method, the frame features to be embedded in every video frame are generated from the frame residuals CS measurements and scrambled with other frame features as marked data. The marked data is embedded into the corresponding frames by modulating color-triples for its least impacts on the carriers. For the receiver, the extracted data is used to reconstruct residuals to conceal errors. Error positions are located using the set theory. Since the CS has the ability to sample a signal within a lower sampling rate than the Shannon–Nyquist rate, the original signal could be reconstructed very well in theory. This indicates that the proposed method could benefit from the CS, and therefore keep better error concealment behavior. The experimental results show that the PSNR values gain about 10 dB averagely and the proposed scheme in this paper improves the video quality significantly comparing with the exiting VECDH schemes.