Error concealment techniques for digital TV

Compressed video bitstreams are intended for real-time transmission over communication networks. Most of the video coding standards employ the temporal and spatial prediction structure to reduce the transmitted video data. Therefore, the coded video bitstreams are highly sensitive to information loss and channel errors. Even a single bit error can lead to disastrous quality degradation in both time and space. This quality deterioration is exacerbated when no error resilient coding mechanism is employed to protect coded video data against the error prone environments. Error concealment is a data recovery technique that enables the decoder to conceal effects of transmission errors by predicting the lost or corrupted video data from the previously reconstructed error-free information. Motion vector recovery and motion compensation with the estimated motion vector is a good approach to conceal the corrupted macroblock data. In this paper, we develop various error concealment algorithms based on motion vector recovery, and compare their performances to those of conventional error concealment methods.     

Spatial shape error concealment for object-based image and video coding

In this paper, an original spatial shape error-concealment technique, to be used in the context of object-based image and video coding schemes, is proposed. In this technique, it is assumed that the shape of the corrupted object at hand is in the form of a binary alpha plane, in which some of the shape data is missing due to channel errors. From this alpha plane, a contour corresponding to the border of the object can be extracted. However, due to errors, some parts of the contour will be missing and, therefore, the contour will be broken. The proposed technique relies on the interpolation of the missing contours with Bézier curves, which is done based on the available surrounding contours. After all the missing parts of the contour have been interpolated, the concealed alpha plane can be easily reconstructed from the fully recovered contour and used instead of the erroneous one improving the final subjective impact.     

An effective GM/LM-based video error concealment

Error concealment (EC) techniques are often utilized at decoder side to improve reconstructed videos, in case of some information is lost during transmission on a wireless or band-width limited communication channel. In this paper, a global motion/local motion (GM/LM)-based error concealment method is proposed. First, the correct-macro-blocks (CMBs) are classified into global motion compensated MBs and local motion compensated MBs adaptively. Then, an erroneous MB (EMB) is classified into one of the three types: global motion MB (GMB), local motion MB (LMB), and global/local overlapping MB (GLMB) according to the MB type information in its neighbors. For the EMB with its type, GMB is recovered using the global motion vector (GMV). The EMB with its type LMB is recovered using the average motion vector information. And for the MB with its type GLMB, a recursive boundary matching strategy is utilized to search an optimal recovering motion vector. Experimental results show the effectiveness of the proposed GM/LM-based error concealment method.     

Spatial error concealment with sequence-aligned texture modeling and adaptive directional recovery

A spatial error concealment technique based on the sequence-aligned texture modeling and the adaptive directional recovery is proposed in this work. The sequence alignment technique captures the local variation and the global trend of image textures with surrounding uncorrupted pixels, and provides the best texture model under a given cost function. With the derived texture model, geometric interpolation is used to recover lost pixels adaptively based on pixel locations. There are four candidate modes of pixel sequences to recover lost pixels, and one of them is selected for the concealment purpose. The selection criterion is based on the texture pattern modes of surrounding uncorrupted blocks. The pixel sequences used for error concealment can be obtained from the computation of the decoder or the side information from the encoder. Extensive experimental results are given to demonstrate that our error concealment technique outperforms several benchmark methods in both objective and subjective tests.     

Adaptive Pixel Interpolation for Spatial Error Concealment

Error concealment techniques are widely used as efficient ways to recover the lost information at the decoder. This paper proposes an adaptive pixel interpolation technique for spatial error concealment in the block based coding system. For a missing pixel in a corrupted block, its value is derived from four neighborhoods of the block through interpolation using multiple prediction strategy. The weighting rules of these four neighborhood blocks are carefully designed with regard to three factors, the distance to the missing pixels within the given corrupted block, the percentage of uncorrupted pixels, and the similarity to the given corrupted block. The proposed method works effectively in consecutive block loss situation, which is common in real applications of video transmission. Experimental results show the proposed technique gains more accurate recovery of the missing pixels than the existing schemes.     

视频对象形状错误隐藏技术研究

视频对象(VO)的形状信息对于正确解码至关重要,当VO的形状发生错误时,为了正确解码纹理和运动信息,需要对其进行错误隐藏。主要介绍VO形状错误隐藏的常用技术,并进行对比实验和讨论。结果表明:当相邻VO间运动较小时,时域法比空域法能更好地保持VO的形状;空域法性能稳定,不受VO间的运动影响,当帧间变化较大时,时域法的参考VO失去参考价值,只能选择基于空域的方法。可见,时域法和空域法各有利弊,一个好的解码器应能实现两种错误隐藏,根据具体情况选择或将两者相结合。最后指出形状错误隐藏今后的研究方向和前景展望。     

Segmentation and tracking of moving objects for content-based videocoding

To enable content-based functionalities in video coding, a decomposition of the scene into physical objects is required. Such objects are normally not characterised by homogeneous colour, intensity, or optical flow. Therefore, conventional techniques based on these low-level features cannot perform the desired segmentation. The authors address segmentation and tracking of moving objects and present a new video object plane (VOP) segmentation algorithm that extracts semantically meaningful objects. A morphological motion filter detects physical objects by identifying areas that are moving differently from the background. A new filter criterion is introduced that measures the deviation of the estimated local motion from the synthesised global motion. A two-dimensional binary model is derived for the object of interest and tracked throughout the sequence by a Hausdorff object tracker. To accommodate for rotations and changes in shape, the model is updated every frame by a two-stage method that accounts for rigid and non-rigid moving parts of the object. The binary model then guides the actual VOP extraction, whereby a novel boundary post-processor ensures high boundary accuracy. Experimental results demonstrate the performance of the proposed algorithm     

Hybrid Temporal Error Concealment Methods for Block-Based Compressed Video Transmission

Boundary matching algorithm (BMA) and decoder motion vector estimation (DMVE) are two well-known temporal error concealment methods using the matching-based approach. In these two methods, the motion vector of each missing block is estimated by choosing one among candidate motion vectors which minimizes a sum of absolute differences (SAD) between boundary pixels of the corrupted macroblock. In general, the performance of DMVE is better than that of BMA. However, depending on the location or pattern of the corrupted block, BMA produces higher visual quality than DMVE. In this paper, we propose two types of hybrid error concealment methods; switching method and blending method. The switching method chooses one of two results obtained by BMA and DMVE based on the normalized SAD values. In the blending method, the weighted sum of the results concealed by the aforementioned two methods is utilized to improve the performance of error concealment. In order to reduce blocking artifacts further, the modified overlapped-block motion compensation is adaptively applied to the concealed blocks. Simulation results show that the proposed methods outperform other techniques in terms of subjective visual quality as well as PSNR performance.     

一种基于对象几何形状的视频抗误码方法

针对压缩视频流中的误码,提出了一种新颖的基于对象几何形状的视频抗误码方法.首先,在编码端通过比较帧间编码帧内邻近宏块的运动向量差异获得粗略的对象几何形状信息,以对象标示图表示;再把它以与标准码流兼容的方式嵌入到压缩码流里去;最后,在解码端利用它帮助精确的构造错误宏块的运动向量.试验结果表明,该方法对于即使大量连续slice出错的情况仍能获得比较好的掩盖效果,而增加的运算复杂度和传输负载量都不大.     

A joint encoder–decoder error control framework for stereoscopic video coding

Stereoscopic video coding (SSVC) plays an important role in various 3D video applications. In SSVC, robust stereoscopic video transmission over error-prone networks is still a challenge problem to be solved. In this paper, we propose a joint encoder–decoder error control framework for SSVC, where error-resilient source coding, transmission network conditions, and error concealment scheme are jointly considered to achieve better error robustness performance. The proposed joint encoder–decoder error control framework includes two parts: an error concealment algorithm at the decoder side and a rate–distortion optimized error resilience algorithm at the encoder side. For error concealment at the decoder side, an overlapped block motion and disparity compensation based error concealment scheme is proposed to adaptively utilize inter-view correlations and temporal correlations. For error resilience at the encoder side, first, the inter-view refreshment is proposed for SSVC to suppress error propagations. Then, an end-to-end distortion model for SSVC is derived, which jointly considers the transmission network conditions, inter-view refreshment, and error concealment tools at the decoder side. Finally, based on the derived end-to-end distortion model, the rate–distortion optimized error resilience algorithm is presented to adaptively select inter-view, inter- or intra-coding for SSVC. The experimental results show that the proposed joint encoder–decoder error control framework has superior error robustness performance for stereoscopic video transmission over error-prone networks.     

Recovery of motion vectors for error concealment based on an edge-detection approach

A new approach to error concealment is presented which exploits both spatial and temporal information from the current and previous frames. The technique consists of two stages, motion vector estimation and enhancement of the estimated motion vector. In the first stage the proposed method estimates a replacement for the corrupted motion vector by applying dynamic weights to related motion vectors from the top, bottom, left and right sub-macroblocks. The estimated motion vectors are then enhanced using a new approach based on edge detection in the second stage. The experimental results for several test video sequences are compared with conventional error concealment methods and higher performance is achieved in both objective peak signal-to-noise ratio measurements and subjective visual quality.     

Motion compensated shape error concealment.

The introduction of Video Objects (VOs) is one of the innovations of MPEG-4. The alpha-plane of a VO defines its shape at a given instance in time and hence determines the boundary of its texture. In packet-based networks, shape, motion, and texture are subject to loss. While there has been considerable attention paid to the concealment of texture and motion errors, little has been done in the field of shape error concealment. In this paper we propose a post-processing shape error concealment technique that uses the motion compensated boundary information of the previously received alpha-plane. The proposed approach is based on matching received boundary segments in the current frame to the boundary in the previous frame. This matching is achieved by finding a maximally smooth motion vector field. After the current boundary segments are matched to the previous boundary, the missing boundary pieces are reconstructed by motion compensation. Experimental results demonstrating the performance of the proposed motion compensated shape error concealment method, and comparing it with the previously proposed weighted side matching method are presented.     

Temporal error concealment using motion field interpolation

An error concealment algorithm based on motion field interpolation is presented. For each pixel in a damaged block, the algorithm recovers a motion vector using bilinear interpolation of neighbouring motion vectors. This vector is then used to conceal the damaged pixel. Overlapped motion compensation is used to combine this algorithm with a boundary matching error concealment algorithm. Simulation results demonstrate the superior performance of the proposed algorithms     

Two-dimensional reversible data hiding-based approach for intra-frame error concealment in H.264/AVC

Highly compressed video bit-stream is extremely sensitive to transmission error. A novel two-dimensional reversible data hiding-based approach for intra-frame error concealment is proposed, which aims at improving video quality at decoder when video bit-stream incur transmission errors. The scheme involves embedding the motion vector (MV) of a macroblock (MB) into other MB within the same intra-frame, and extracting the embedded MV from the received video frame for reconstruction of the corrupted MB. Based on the distribution of the motion vector data and the characteristic of histogram shifting, a specific two-dimensional reversible data hiding mechanism is designed. Consequently, the distortion of the marked video can be controlled at a low level. Experimental results show that the damaged macroblocks can be recovered with a higher quality using the reversible data hiding methodology, as compared to the non-reversible data hiding method. Furthermore, experimental results demonstrate that the proposed algorithm outperforms some state-of-the-art reversible data hiding error concealment schemes in improving the perceptual quality.     

An improved reversible data hiding-based approach for intra-frame error concealment in H.264/AVC

As H.264/AVC video streams are highly compressed, they become sensitive to errors caused by unreliable transmission channels. In order to address this issue, an improved version of Chung et al.’s reversible data hiding-based approach for intra-frame error concealment is proposed for H.264/AVC codec. By using the histogram shifting technique, the original work reversibly embeds the motion vector (MV) of a macroblock (MB) into other MB within the same intra-frame. If an MB is corrupted at the decoder side, the embedded MV can be extracted from the corresponding MB for the recovery of the corrupted MB. However, Chung et al.’s work did not fully exploit the number of coefficients which need to be modified in order to reversibly hiding data, and did not consider many extra nonzero residual blocks produced by data hiding. These two issues could reduce the visual quality of the stego-video. This paper adopts MV data pre-processing, the selection of most suitable embedding region, and the minimum possible amount of histogram modification, which lead to higher PSNR of the stego-video for a given payload. Experimental results further reveal that the proposed method offers stego-video with better visual quality over Chung et al.’s work.     

Motion field interpolation for temporal error concealment

When transmitted over practical communication channels, compressed video can suffer severe degradation. One approach to combat the effect of channel errors is error concealment. It is an attractive choice because it does not increase the bit rate, it does not require any modifications to the encoder, it does not introduce any delays and it can be applied in almost any application. Conventional temporal concealment techniques estimate one concealment displacement for the whole damaged block and then use translational displacement compensation to conceal the block from a reference frame. The main problem with such techniques is that incorrect estimation of the concealment displacement can lead to poor concealment of the whole or most of the block. Two novel temporal concealment techniques are presented. In the first technique, motion field interpolation is used to estimate one concealment displacement per pel of the damaged block and then each pel is concealed individually. In this case, incorrect estimation of a concealment displacement will only affect the corresponding pel. On a block level this may affect few pels rather than the entire block. In the second technique, multi-hypothesis motion compensation is used to combine the first technique with a boundary matching temporal concealment technique to obtain a more robust performance. Simulation results, within both an isolated error propagation environment and an H.263 codec, show the superior subjective and objective performance of the proposed techniques when compared with conventional temporal concealment techniques     

Joint source-channel coding for wireless object-based video communications utilizing data hiding.

In recent years, joint source-channel coding for multimedia communications has gained increased popularity. However, very limited work has been conducted to address the problem of joint source-channel coding for object-based video. In this paper, we propose a data hiding scheme that improves the error resilience of object-based video by adaptively embedding the shape and motion information into the texture data. Within a rate-distortion theoretical framework, the source coding, channel coding, data embedding, and decoder error concealment are jointly optimized based on knowledge of the transmission channel conditions. Our goal is to achieve the best video quality as expressed by the minimum total expected distortion. The optimization problem is solved using Lagrangian relaxation and dynamic programming. The performance of the proposed scheme is tested using simulations of a Rayleigh-fading wireless channel, and the algorithm is implemented based on the MPEG-4 verification model. Experimental results indicate that the proposed hybrid source-channel coding scheme significantly outperforms methods without data hiding or unequal error protection.     

Motion Compensated Three-Dimensional Frequency Selective Extrapolation for improved error concealment in video communication

During transmission of video data over error-prone channels the risk of getting severe image distortions due to transmission errors is ubiquitous. To deal with image distortions at decoder side, error concealment is applied. This article presents Motion Compensated Three-Dimensional Frequency Selective Extrapolation, a novel spatio-temporal error concealment algorithm. The algorithm uses fractional-pel motion estimation and compensation as initial step, being followed by the generation of a model of the distorted signal. The model generation is conducted by an enhanced version of Three-Dimensional Frequency Selective Extrapolation, an existing error concealment algorithm. Compared to this existent algorithm, the proposed one yields an improvement in concealment quality of up to 1.64 dB PSNR. Altogether, the incorporation of motion compensation and the improved model generation extends the already high extrapolation quality of the underlying Frequency Selective Extrapolation, resulting in a gain of more than 3 dB compared to other well-known error concealment algorithms.     

Shape error concealment using Hermite splines

The introduction of video objects (VOs) is one of the innovations of MPEG-4. The alpha-plane of a VO defines its shape at a given instance in time and hence determines the boundary of its texture. In packet-based networks, shape, motion, and texture are subject to loss. While there has been considerable attention paid to the concealment of texture and motion errors, little has been done in the field of shape error concealment. In this paper, we propose a post-processing shape error-concealment technique that uses geometric boundary information of the received alpha-plane. Second-order Hermite splines are used to model the received boundary in the neighboring blocks, while third order Hermite splines are used to model the missing boundary. The velocities of these splines are matched at the boundary point closest to the missing block. There exists the possibility of multiple concealing splines per group of lost boundary parts. Therefore, we draw every concealment spline combination that does not self-intersect and keep all possible results until the end. At the end, we select the concealment solution that results in one closed boundary. Experimental results demonstrating the performance of the proposed method and comparisons with prior proposed methods are presented.     

A generic framework for tracking using particle filter with dynamic shape prior.

Tracking deforming objects involves estimating the global motion of the object and its local deformations as functions of time. Tracking algorithms using Kalman filters or particle filters (PFs) have been proposed for tracking such objects, but these have limitations due to the lack of dynamic shape information. In this paper, we propose a novel method based on employing a locally linear embedding in order to incorporate dynamic shape information into the particle filtering framework for tracking highly deformable objects in the presence of noise and clutter. The PF also models image statistics such as mean and variance of the given data which can be useful in obtaining proper separation of object and background.