Dictionary learning based reconstruction for distributed compressed video sensing

Distributed compressed video sensing (DCVS) is a framework that integrates both compressed sensing and distributed video coding characteristics to achieve a low-complexity video coding. However, how to design an efficient reconstruction by leveraging more realistic signal models that go beyond simple sparsity is still an open challenge. In this paper, we propose a novel “undersampled” correlation noise model to describe compressively sampled video signals, and present a maximum-likelihood dictionary learning based reconstruction algorithm for DCVS, in which both the correlation and sparsity constraints are included in a new probabilistic model. Moreover, the signal recovery in our algorithm is performed during the process of dictionary learning, instead of being employed as an independent task. Experimental results show that our proposal compares favorably with other existing methods, with 0.1–3.5 dB improvements in the average PSNR, and a 2–9 dB gain for non-key frames when key frames are subsampled at an increased rate.     

Prediction of transmission distortion for wireless video communication: Algorithm and application

In this paper, we develop algorithms for estimating transmission distortion in wireless video communication systems. By leveraging the analytical results obtained in our previous works, we design low complexity algorithms that are capable of estimating transmission distortion accurately. We also extend our algorithm for pixel-level transmission distortion estimation to pixel-level end-to-end distortion estimation. Furthermore, we apply our pixel-level end-to-end distortion estimation algorithm to prediction mode decision in H.264 encoder. Experimental results show that (1) our transmission distortion estimation algorithm is more accurate and more robust against inaccurate channel estimation than existing distortion estimation algorithms; (2) our mode decision algorithm achieves remarkable PSNR gain over the existing algorithms for prediction mode decision in H.264 encoder, e.g., an average PSNR gain of 1.44 dB for ‘foreman’ sequence when Packet Error Probability (PEP) equals 5%.     

Correlation estimation from compressed images

This paper addresses the problem of correlation estimation in sets of compressed images. We consider a framework where the images are represented under the form of linear measurements due to low complexity sensing or security requirements. We assume that the images are correlated through the displacement of visual objects due to motion or viewpoint change and the correlation is effectively represented by optical flow or motion field models. The correlation is estimated in the compressed domain by jointly processing the linear measurements. We first show that the correlated images can be efficiently related using a linear operator. Using this linear relationship we then describe the dependencies between images in the compressed domain. We further cast a regularized optimization problem where the correlation is estimated in order to satisfy both data consistency and motion smoothness objectives with a Graph Cut algorithm. We analyze in detail the correlation estimation performance and quantify the penalty due to image compression. Extensive experiments in stereo and video imaging applications show that our novel solution stays competitive with methods that implement complex image reconstruction steps prior to correlation estimation. We finally use the estimated correlation in a novel joint image reconstruction scheme that is based on an optimization problem with sparsity priors on the reconstructed images. Additional experiments show that our correlation estimation algorithm leads to an effective reconstruction of pairs of images in distributed image coding schemes that outperform independent reconstruction algorithms by 2–4 dB.     

A fine-grain distortion and complexity aware parameter tuning model for the H.264/AVC encoder

Most existing video encoders currently used in mobile applications are unable to gracefully degrade their output quality as the battery life nears its end. In other words, they cannot manage power consumption to efficiently utilize the available power resources. To be able to effectively adapt to changes in the encoder′s software and hardware platforms, especially due to the power limitations of mobile devices, the effect of encoder parameters on the encoding quality and power consumption has to be represented using a Rate–Distortion–Complexity (R–D–C) model. Most existing R–D–C models only consider macroblock level parameters, and overlook other higher level parameters that may have a more significant impact on complexity. In this paper, the distortion and complexity of the H.264/AVC encoder is controlled considering a subset of higher level encoding parameters consisting of search range, number of reference frames, and motion vector resolution. First, the complexity of full and fast motion estimation methods is modeled in an implementation and platform independent manner. Then, using this complexity model, a common encoding parameter setting table is derived, which leads to the least amount of distortion for each complexity condition. Finally, a complexity control mechanism is proposed which tunes the encoding parameters in a real-time manner. The proposed model can be combined with other existing macroblock level models in order to design a two-phase fine grain complexity controller. Simulation results indicate that when our method is integrated with the direct resource allocation (DRA) approach, performance increases by an average of 1.02 dB and 1.06 dB for full and fast motion estimation approaches, respectively.     

Motion Compensated Three-Dimensional Frequency Selective Extrapolation for improved error concealment in video communication

During transmission of video data over error-prone channels the risk of getting severe image distortions due to transmission errors is ubiquitous. To deal with image distortions at decoder side, error concealment is applied. This article presents Motion Compensated Three-Dimensional Frequency Selective Extrapolation, a novel spatio-temporal error concealment algorithm. The algorithm uses fractional-pel motion estimation and compensation as initial step, being followed by the generation of a model of the distorted signal. The model generation is conducted by an enhanced version of Three-Dimensional Frequency Selective Extrapolation, an existing error concealment algorithm. Compared to this existent algorithm, the proposed one yields an improvement in concealment quality of up to 1.64 dB PSNR. Altogether, the incorporation of motion compensation and the improved model generation extends the already high extrapolation quality of the underlying Frequency Selective Extrapolation, resulting in a gain of more than 3 dB compared to other well-known error concealment algorithms.     

Fuzzy quantization based bit transform for low bit-resolution motion estimation

This study proposes a novel fuzzy quantization based bit transform for low bit-resolution motion estimation. We formalize the procedure of bit resolution reduction by two successive steps, namely interval partitioning and interval mapping. The former is a many-to-one mapping which determines motion estimation performance, while the latter is a one-to-one mapping. To gain a reasonable interval partitioning, we propose a non-uniform quantization method to compute coarse thresholds. They are then refined by using a membership function to solve the mismatch of pixel values near threshold caused by camera noise, coding distortion, etc. Afterwards, we discuss that the sum of absolute difference (SAD) is one of the fast matching metrics suitable for low bit-resolution motion estimation in the sense of mean squared errors. A fuzzy quantization based low bit-resolution motion estimation algorithm is consequently proposed. Our algorithm not only can be directly employed in video codecs, but also be applied to other fast or complexity scalable motion estimation algorithms. Extensive experimental results show that the proposed algorithm can always achieve good motion estimation performances for video sequences with various characteristics. Compared with one-bit transform, multi-thresholding two-bit transform, and adaptive quantization based two-bit transform, our bit transform separately gains 0.98 dB, 0.42 dB, and 0.24 dB improvement in terms of average peak signal-to-noise ratio, with less computational cost as well.     

Video super-resolution based on automatic key-frame selection and feature-guided variational optical flow

This paper proposes a new video super-resolution method based on feature-guided variational optical flow. The key-frames are automatically selected and super-resolved using a method based on sparse regression. To overcome the blocking artifacts and deal with the case of small structures with large displacement, an efficient method based on feature-guided variational optical flow is used to super-resolve the non-key-frames. Experimental results show that the proposed method outperforms the existing benchmark in terms of both subjective visual quality and objective peak signal-to-noise ratio (PSNR). The average PSNR improvement from the bi-cubic interpolation is 7.15 dB for four datasets. Thanks to the flexibility of designed automatic key-frame selection and the validness of feature-guided variational optical flow, the proposed method is applicable to various practical video sequences.     

Complexity-based intra frame rate control by jointing inter-frame correlation for high efficiency video coding

Rate control is of great significance for the High Efficiency Video Coding (HEVC). Due to the high efficiency and low complexity, the R-lambda model has been applied to the HEVC as the default rate control algorithm. However, the video content complexity, which can help improve the code efficiency and rate control performance, is not fully considered in the R-lambda model. To address this problem, an intra-frame rate control algorithm, which aims to provide improved and smooth video quality, is developed in this paper by jointly taking into consideration the frame-level content complexity between the encoded intra frames and the encoded inter frame, as well as the CTU-level complexity among different CTUs in texture–different regions for intra-frame. Firstly, in order to improve the rate control efficiency, this paper introduces a new prediction measure of content complexity for CTUs of intra-frame by jointly considering the inter-frame correlations between encoding intra frame and previous encoded inter frames as well as correlations between encoding intra frame and previous encoded intra frame. Secondly, a frame-level complexity-based bit-allocation-balancing method, by jointly considering the inter-frame correlation between intra frame and previous encoded inter frame, is brought up so that the smoothness of the visual quality can be improved between adjacent inter- and intra-frames. Thirdly, a new region-division and complexity-based CTU-level bit allocation method is developed to improve the objective quality and to reduce PSNR fluctuation among CTUs in intra-frame. In the end, related model parameters are updated during the encoding process to increase rate control accuracy. As a result, as can be seen from the extensive experimental results that compared with the state-of-the-art schemes, the video quality can be significantly improved. More specifically, up to 10.5% and on average 5.2% BD-Rate reduction was achieved compared to HM16.0 and up to 2.7% and an average of 2.0% BD-Rate reduction was achieved compared to state-of-the-art algorithm. Besides, a superior performance in enhancing the smoothness of quality can be achieved, which outperforms the state-of-the-art algorithms in term of flicker measurement, frame and CTU-wise PSNR, as well as buffer fullness.     

利用视频非局部相似性的分布式压缩感知重构

为了提高分布式视频压缩感知（Distributed Video Compressive Sensing,DVCS）的率失真性能,仅利用稀疏先验知识不能很好地保护视频帧的边缘与纹理细节,本文提出利用视频非局部相似性形成正则化项融入联合重构模型以有效去除边缘与纹理区域的模糊和块效应现象。仿真实验表明,本文所提出的联合重构算法可有效地改善主客观视频重构质量,能以一定计算复杂度为代价提高分布式视频压缩感知系统的率失真性能。      

A 3D-DCT video encoder using advanced coding techniques for low power mobile device

The three-dimensional discrete cosine transform (3D-DCT) has been researched as an alternative to existing dominant video standards based on motion estimation and compensation. Since it does not need to search macro block for inter/intra prediction, 3D-DCT has great advantages for complexity. However, it has not been developed well because of poor video quality while video standards such as H.263(+) and HEVC have been blooming. In this paper, we propose a new 3D-DCT video coding as a new video solution for low power mobile technologies such as Internet of Things (IoT) and Drone. We focus on overcoming drawbacks reported in previous research. We build a complete 3D-DCT video coding system by adopting existing advanced techniques and devising new coding algorithms to improve overall performance of 3D-DCT. Experimental results show proposed 3D-DCT outperforms H.264 low power profiles while offering less complexity. From GBD-PSNR, proposed 3D-DCT provides better performance by average 4.6 dB.     

Efficient reconstruction of compressively sensed images and videos using non-iterative method

Compressed sensing is widely applied for compression and reconstruction of images and videos by projecting the pixel values to smaller dimensional measurements. These measurements are reconstructed at the receiver using various reconstruction procedures. Greedy algorithms are often used for such recovery. These solve the least squares problem to find the best match with minimum error. This is a time consuming and complex process, giving rise to a trade-off between reconstruction performance and algorithmic performance. This work proposes a non-iterative method, viz., non-iterative pseudo inverse based recovery algorithm (NIPIRA), for reconstruction of compressively sensed images and videos with small complexity and time consumption, provided the reconstruction quality is maintained. NIPIRA gives a minimum PSNR of 32 dB for very few measurements (M/N = 0.3125) and accuracy of above 97%. There is more than 92% of decrease in elapsed time compared with other iterative algorithms. NIPIRA is tested for its performance with respect to many other objective measures as well. The complexity of NIPIRA is s times less than existing recovery algorithms.     

Optimal-correlation-based reconstruction for distributed compressed video sensing

Distributed compressed video sensing (DCVS) is a framework that integrates both compressed sensing and distributed video coding characteristics to achieve a low-complexity video coding. However, how to design an efficient joint reconstruction by leveraging more realistic signal models is still an open challenge. In this paper, we present a novel optimal-correlation-based reconstruction method for compressively sampled videos from multiple measurement vectors. In our method, the sparsity is mainly exploited through inter-signal correlations rather than the traditional frequency transform, wherein the optimization is not only over the signal space to satisfy data consistency but also over all possible linear correlation models to achieve minimum-l₁-norm correlation noise. Additionally, a two-phase Bregman iterative based algorithm is outlined for solving the optimization problem. Simulation results show that our proposal can achieve an improved reconstruction performance in comparison to the conventional approaches, and especially, offer a 0.7–9.9 dB gain in the average PSNR for DCVS.     

Image representation by compressive sensing for visual sensor networks

This paper addresses the image representation problem in visual sensor networks. We propose a new image representation method for visual sensor networks based on compressive sensing (CS). CS is a new sampling method for sparse signals, which is able to compress the input data in the sampling process. Combining both signal sampling and data compression, CS is more capable of image representation for reducing the computation complexity in image/video encoder in visual sensor networks where computation resource is extremely limited. Since CS is more efficient for sparse signals, in our scheme, the input image is firstly decomposed into two components, i.e., dense and sparse components; then the dense component is encoded by the traditional approach (JPEG or JPEG 2000) while the sparse component is encoded by a CS technique. In order to improve the rate distortion performance, we leverage the strong correlation between dense and sparse components by using a piecewise autoregressive model to construct a prediction of the sparse component from the corresponding dense component. Given the measurements and the prediction of the sparse component as initial guess, we use projection onto convex set (POCS) to reconstruct the sparse component. Our method considerably reduces the number of random measurements needed for CS reconstruction and the decoding computational complexity, compared to the existing CS methods. In addition, our experimental results show that our method may achieves up to 2 dB gain in PSNR over the existing CS based schemes, for the same number of measurements.     

An automatic region-based video sequence codec based on fractal compression

In order to improve the performance of fractal video coding, we explore a novel fractal video sequences codec with automatic region-based functionality. To increase the quality of decoding image, intra frame coding, deblocking loop filter and sub-pixel block matching are applied to the codec. An efficient searching algorithm is used to increase the compression ratio and encoding speed. Automatic region-based fractal video sequences coding reduces coding stream greatly. Experimental results indicate that the proposed algorithm is more robust, and provides much less encoding time and bitrate while maintaining the quality of decompression image than the conventional CPM/NCIM method and other related references. We compare the proposed algorithm with three algorithms in Refs. [24], [25], [26], and the results of all these four algorithms are compared with H.264. The bitrate of the proposed algorithm is decreased by 0.11% and the other algorithms are increased by 4.29%, 6.85% and 11.62%, respectively. The average PSNR degradations of the four algorithms are 0.71 dB, 0.48 dB, 0.48 dB and 0.75 dB. So the bitrate of the proposed algorithm is decreased and the other algorithms are increased. At the meantime the compression time is reduced greatly, about 79.19% on average. The results indicate that, on average, the proposed automatic region-based fractal video sequences coding system can save compression time 48.97% and bitrate 52.02% with some image quality degradation in comparison with H.264, since they are all above 32 dB and the human eyes are insensitive to the differences.     

基于自适应PCA稀疏基底的分布式视频压缩感知重构

为了提高分布式视频压缩感知（Distributed Video Compressive Sensing,DVCS）的率失真性能,本文提出利用自适应稀疏基底进行联合重构。提出算法利用帧间运动信息形成样本数据矩阵,再利用主成份分析（Principle Components Analysis,PCA）训练出其显著主成份构成稀疏字典,该稀疏字典不仅可根据视频时空统计特征自适应变化而且可有效地抑制噪声。仿真实验表明,本文所提出的联合重构算法可有效地改善主客观视频重构质量,能够以一定的计算复杂度为代价提高DVCS系统的率失真性能。     

Advanced side information creation techniques and framework for Wyner–Ziv video coding

Recently, several distributed video coding (DVC) solutions based on the distributed source coding (DSC) paradigm have appeared in the literature. Wyner–Ziv (WZ) video coding, a particular case of DVC where side information is made available at the decoder, enable to achieve a flexible distribution of the computational complexity between the encoder and decoder, promising to fulfill novel requirements from applications such as video surveillance, sensor networks and mobile camera phones. The quality of the side information at the decoder has a critical role in determining the WZ video coding rate-distortion (RD) performance, notably to raise it to a level as close as possible to the RD performance of standard predictive video coding schemes. Towards this target, efficient motion search algorithms for powerful frame interpolation are much needed at the decoder. In this paper, the RD performance of a Wyner–Ziv video codec is improved by using novel, advanced motion compensated frame interpolation techniques to generate the side information. The development of these type of side information estimators is a difficult problem in WZ video coding, especially because the decoder only has available some reference, decoded frames. Based on the regularization of the motion field, novel side information creation techniques are proposed in this paper along with a new frame interpolation framework able to generate higher quality side information at the decoder. To illustrate the RD performance improvements, this novel side information creation framework has been integrated in a transform domain turbo coding based Wyner–Ziv video codec. Experimental results show that the novel side information creation solution leads to better RD performance than available state-of-the-art side information estimators, with improvements up to 2 dB; moreover, it allows outperforming H.264/AVC Intra by up to 3 dB with a lower encoding complexity.     

Novel rate control scheme for intra frame video coding with exponential rate–distortion model on H.264/AVC

Rate control regulates the output bit rate of a video encoder in order to obtain optimum visual quality within the available network bandwidth and to maintain buffer fullness within a specified tolerance range. Due to the benefits of intra-only encoding, such as less computational cost and less latency, it has been more and more widely used. In this paper, we propose an accurate intra-only rate control scheme for H.264/AVC, which includes a novel complexity measurement and a new rate–distortion (R–D) model. We also propose a linear rate–complexity model which takes the intercept into consideration to reduce the estimation error. The proposed R–D model is integrated by the linear rate–complexity model and an exponential rate–quantization model. Based on theoretical analysis and experimental validation, the proposed scheme has high bits prediction precision, and it can also accurately handle buffer fullness. Compared with JVT-W042, our algorithm achieves higher average PSNR and improves the coding quality up to 0.35 dB.     

Channel estimation with ISFLA based pilot pattern optimization for MIMO OFDM system

In multiple input multiple output orthogonal frequency division multiplexing (MIMO-OFDM) systems, the channel state information should be known by the receiver for obtaining transmitted data. Channel estimation algorithms are used to examine the multipath effects of frequency selective Rayleigh fading channels. In this paper, Compressed Sensing (CS) based channel estimation technique is considered for reconstructing the signal with improved spectral efficiency. It requires transmitting the known pilot data to the receiver for estimating channel information. The optimum pilot patterns are selected through reducing the mutual coherence of measurement matrix. In order to maximize the accuracy of sparse channel estimation and to reduce the computational complexity, an optimization algorithm Improved Shuffled Frog Leaping (ISFL) is proposed. When compared with the traditional estimation methods like least squares (LS), and minimal mean square error (MMSE), 4.7% of spectral efficiency is increased with ISFLA based channel estimation. Implementation results show that, by using the proposed algorithm, the bit error rate (BER) and Mean Square Error (MER) performance of the system is increased with 1.5 dB and 2 dB respectively.     

View-spatial–temporal post-refinement for view synthesis in 3D video systems

Depth image based rendering is one of key techniques to realize view synthesis for three-dimensional television and free-viewpoint television, which provide high quality and immersive experiences to end viewers. However, artifacts of rendered images, including holes caused by occlusion/disclosure and boundary artifacts, may degrade the subjective and objective image quality. To handle these problems and improve the quality of rendered images, we present a novel view-spatial–temporal post-refinement method for view synthesis, in which new hole-filling and boundary artifact removal techniques are proposed. In addition, we propose an optimal reference frame selection algorithm for a better trade-off between the computational complexity and rendered image quality. Experimental results show that the proposed method can achieve a peak signal-to-noise ratio gain of 0.94 dB on average for multiview video test sequences when compared with the benchmark view synthesis reference software. In addition, the subjective quality of the rendered image is also improved.     

Fast transmission distortion estimation and adaptive error protection for H.264/AVC-based embedded video conferencing systems

This paper presents the design and implementation of an error-resilient H.264/AVC-based embedded video conferencing scheme over Internet. We first develop a fast recursive algorithm to estimate the decoder-side distortion of each frame in the presence of packet loss. The algorithm operates at block level, and considers the impacts of different intra prediction modes, the unrestricted intra prediction, and the skip mode. We then design a family of very short systematic forward error correction codes with linear encoding and decoding complexity, which runs across the slices of each frame to recover lost packets. An optimization problem is then formulated to minimize the decoder-side distortion by allocating a given channel coding redundancy among a group of frames. Various techniques are introduced to speed up the algorithm without sacrificing too much accuracy, in order to meet the hardware and real-time constraints of the system. As a result, the proposed scheme can run on a real-time embedded video conferencing system with resolution up to 1024×576 pixels, 30 frames per second (fps) and 4 megabits per second (Mbps).