Pixel-wise video stabilization

In this paper, we present a novel video stabilization method with a pixel-wise motion model. In order to avoid distortion introduced by traditional feature points based motion models, we focus on constructing a more accurate model to capture the motion in videos. By taking advantage of dense optical flow, we can obtain the dense motion field between adjacent frames and set up a pixel-wise motion model which is accurate enough. Our method first estimates dense motion field between adjacent frames. A PatchMatch based dense motion field estimation algorithm is proposed. This algorithm is specially designed for similar video frames rather than arbitrary images to reach higher speed and better performance. Then, a simple and fast smoothing algorithm is performed to make the jittered motion stabilized. After that, we warp input frames using a weighted average algorithm to construct the output frames. Some pixels in output frames may be still empty after the warping step, so in the last step, these empty pixels are filled using a patch based image completion algorithm. We test our method on many challenging videos and demonstrate the accuracy of our model and the effectiveness of our method.     

噪声条件下基于粒子群优化的数字稳像方法*

当视频序列中同时存在随机噪声和随机晃动时,传统的数字稳像算法由于受到噪声干扰而无法有效消除视频序列中的随机晃动。为了稳定这种存在随机噪声的视频序列,提出了一种基于粒子群优化的数字稳像方法。首先,定义了衡量寻优结果适应度函数,即输入视频连续若干帧均值图像的能量;然后,算法利用粒子群优化策略来搜索视频序列的最优运动补偿向量;最后,实验分别使用模拟抖动视频和真实拍摄的视频来测试算法的性能。实验结果表明,当测试视频同时存在随机噪声和随机晃动时,该算法不仅能够有效消除视频的随机晃动,并且有效抑制了随机噪声。     

Multiplane Video Stabilization

This paper presents a novel video stabilization approach by leveraging the multiple planes structure of video scene to stabilize inter‐frame motion. As opposed to previous stabilization procedure operating in a single plane, our approach primarily deals with multiplane videos and builds their multiple planes structure for performing stabilization in respective planes. Hence, a robust plane detection scheme is devised to detect multiple planes by classifying feature trajectories according to reprojection errors generated by plane induced homographies. Then, an improved planar stabilization technique is applied by conforming to the compensated homography in each plane. Finally, multiple stabilized planes are coherently fused by content‐preserving image warps to obtain the output stabilized frames. Our approach does not need any stereo reconstruction, yet is able to produce commendable results due to awareness of multiple planes structure in the stabilization. Experimental results demonstrate the effectiveness and efficiency of our approach to robust stabilization on multiplane videos.     

Motion-estimation-based stabilization of infrared video

In the course of the filming of infrared (IR) video, intrinsic equipment instability incurs movement that in turn causes image blurring. For image clarity and viewing comfortability, it is required that such movement be countered. Presently, video stabilization systems perform Motion Estimation of frames that is then applied frame-by-frame to subsequent frames in order to calculate a motion vector, counter movement, and produce, thereby, a more stable image. However, frame-by-frame comparison for long-distance filming often is difficult due to lack of information. The present study determined the appropriate blocks with the most information for Motion Estimation. We also were able to differentiate between equipment movement and movement in the video itself. By these means, we were able to stabilize videos. The methods employed in the experimentation were 5 sets of 640 × 480 long-distance videos and 5 sets of 480 × 320 long-distance videos. When compared with the current motion estimation methods, our proposed method afforded a 10% increase in accuracy.     

Capturing Intention‐based Full‐Frame Video Stabilization

Annoying shaky motion is one of the significant problems in home videos, since hand shake is an unavoidable effect when capturing by using a hand‐held camcorder. Video stabilization is an important technique to solve this problem, but the stabilized videos resulting from some current methods usually have decreased resolution and are still not so stable. In this paper, we propose a robust and practical method of full‐frame video stabilization while considering user's capturing intention to remove not only the high frequency shaky motions but also the low frequency unexpected movements. To guess the user's capturing intention, we first consider the regions of interest in the video to estimate which regions or objects the user wants to capture, and then use a polyline to estimate a new stable camcorder motion path while avoiding the user's interested regions or objects being cut out. Then, we fill the dynamic and static missing areas caused by frame alignment from other frames to keep the same resolution and quality as the original video. Furthermore, we smooth the discontinuous regions by using a three‐dimensional Poisson‐based method. After the above automatic operations, a full‐frame stabilized video can be achieved and the important regions and objects can also be preserved.     

即时全变差优化的低延时视频稳像方法

目的 传统的视频稳像方法为了获得理想的稳像效果,一般耗费较多的计算时间,且存在较长的延时。针对此问题,提出一种即时全变差优化的低延时视频稳像方法。方法 首先利用特征点检测和匹配计算帧间单应变换,得到抖动视频的运动路径;然后通过即时全变差优化方法对抖动路径进行平滑优化,获得稳定的运动路径;最后通过运动补偿,生成稳定的视频。结果 对公共视频数据集中的抖动视频进行稳像效果测试,并与当前稳像效果较好的几种稳像算法和商业软件进行效果和时间对比。在时间方面,统计了不同方法的每帧平均消耗时间和处理延迟帧数,不同于后期处理方法需要得到大部分视频帧才能够进行计算,本文算法能够在只有一帧延时的情况下获得最终的稳像结果,相比于MeshFlow方法有15%左右的速度提升;在稳像效果方面,计算了不同方法稳像后的视频扭曲率和裁剪率,并邀请非专业用户进行了稳定程度的主观判断,本文算法的实验结果并不输于目前被公认较好的3种后期稳像方法,优于Kalman滤波方法。结论 本文所提稳像方法能够兼顾速度和有效性,相对于传统方法,更适合低延时要求的应用场景。     

Single Sensor Compressive Light Field Video Camera

This paper presents a novel compressed sensing (CS) algorithm and camera design for light field video capture using a single sensor consumer camera module. Unlike microlens light field cameras which sacrifice spatial resolution to obtain angular information, our CS approach is designed for capturing light field videos with high angular, spatial, and temporal resolution. The compressive measurements required by CS are obtained using a random color-coded mask placed between the sensor and aperture planes. The convolution of the incoming light rays from different angles with the mask results in a single image on the sensor; hence, achieving a significant reduction on the required bandwidth for capturing light field videos. We propose to change the random pattern on the spectral mask between each consecutive frame in a video sequence and extracting spatio-angular-spectral-temporal 6D patches. Our CS reconstruction algorithm for light field videos recovers each frame while taking into account the neighboring frames to achieve significantly higher reconstruction quality with reduced temporal incoherencies, as compared with previous methods. Moreover, a thorough analysis of various sensing models for compressive light field video acquisition is conducted to highlight the advantages of our method. The results show a clear advantage of our method for monochrome sensors, as well as sensors with color filter arrays.     

Depth Mapping for Stereoscopic Videos

Stereoscopic videos have become very popular in recent years. Most of these videos are developed primarily for viewing on large screens located at some distance away from the viewer. If we watch these videos on a small screen located near to us, the depth range of the videos will be seriously reduced, which can significantly degrade the 3D effects of these videos. To address this problem, we propose a linear depth mapping method to adjust the depth range of a stereoscopic video according to the viewing configuration, including pixel density and distance to the screen. Our method tries to minimize the distortion of stereoscopic image contents after depth mapping, by preserving the relationship of neighboring features and preventing line and plane bending. It also considers the depth and motion coherences. While depth coherence ensures smooth changes of the depth field across frames, motion coherence ensures smooth content changes across frames. Our experimental results show that the proposed method can improve the stereoscopic effects while maintaining the quality of the output videos.     

Video flickering removal using temporal reconstruction optimization

In this paper, we introduce an approach to remove the flickers in the videos, and the flickers are caused by applying image-based processing methods to original videos frame by frame. First, we propose a multi-frame based video flicker removal method. We utilize multiple temporally corresponding frames to reconstruct the flickering frame. Compared with traditional methods, which reconstruct the flickering frame just from an adjacent frame, reconstruction with multiple temporally corresponding frames reduces the warp inaccuracy. Then, we optimize our video flickering method from following aspects. On the one hand, we detect the flickering frames in the video sequence with temporal consistency metrics, and just reconstructing the flickering frames can accelerate the algorithm greatly. On the other hand, we just choose the previous temporally corresponding frames to reconstruct the output frames. We also accelerate our video flicker removal with GPU. Qualitative experimental results demonstrate the efficiency of our proposed video flicker method. With algorithmic optimization and GPU acceleration, the time complexity of our method also outperforms traditional video temporal coherence methods.

     

结合光流法和卡尔曼滤波的视频稳像算法

针对手机拍摄过程中产生的视频抖动问题,提出了一种基于光流法和卡尔曼滤波的视频稳像算法。首先通过光流法预稳定抖动视频,对其生成的预稳定视频帧进行Shi-Tomasi角点检测,并采用LK算法跟踪角点,再利用RANSAC算法估计相邻帧间的仿射变换矩阵,由此计算得出原始相机路径;然后通过卡尔曼滤波器优化平滑相机路径,得到平滑相机路径;最后由原始相机路径与平滑路径的关系,计算相邻帧间的补偿矩阵,再利用补偿矩阵对视频帧逐一进行几何变换,由此得到稳定的视频输出。实验表明,该算法在处理6大类抖动视频时均有较好的效果,其中稳像后视频的PSNR值相比原始视频的PSNR值约提升了6.631 dB,视频帧间的结构相似性SSIM约提升了40%,平均曲率值约提升了8.3%。     

基于随机性检测的数字稳像算法评价方法*

提出了一种基于随机性检测的数字稳像算法客观评价方法。该方法首先估计出数字稳像算法输出视频的全局运动向量;然后将估计结果编码为二进制序列;最后利用随机性检测的方法来检验此二进制序列的随机性,并且根据序列随机性的强弱程度来衡量数字稳像算法的效果。最后的实验表明,提出的方法能够准确地评价常用的数字稳像算法。     

Cylindrical panoramic mosaicing from a pipeline video through MRF based optimization

Stratum structure detection is a fundamental problem in geological engineering. One of the most commonly employed detection technologies is to shoot videos of a borehole using a forward moving camera. Using this technology, the problem of stratum structure detection is transformed into the problem of constructing a panoramic image from a low quality video. In this paper, we propose a novel method for creating a panoramic image of a borehole from a video sequence without the need of camera calibration and tracking. To stitch together pixels of neighboring image frames, our camera model is designed with a focal length changing feature, along with a small rotational freedom in the two-dimensional image space. Our camera model assumes that target objects lie on a cylindrical wall and that the camera moves forward along the central axis of the cylindrical wall. Based on these two assumptions, our method robustly resolves these two degrees-of-freedoms in our camera model through KLT feature tracking. Since the quality of the result video is affected by possible illumination overflow, camera lens blurring, and low video resolution, we introduce a cost function for eliminating seams between stitching strips. Our cost function is designed based on Markov Random Field and optimized using a belief propagation algorithm. Using our method, we can automatically construct a panorama image with good resolution, smoothness, and continuousness both in the texture and illumination space. Experiment results show that our method could efficiently generate panoramas for long video sequences with satisfying visual quality.     

一种内容完整的视频稳定算法

设计了一种基于可靠特征集合匹配的内容完整的视频稳定算法。为了避免运动前景上的特征点参与运动估计,由经典的KLT(Kanade-Lucas-Tomasi)算法提取特征点,而后基于特征有效性判定规则对特征点集合进行有效性验证以提高特征点的可靠性。利用通过验证的特征点对全局运动进行估计,得到精确的运动参数并据此对视频图像进行运动补偿。对于运动补偿造成的无定义区,首先计算当前帧的定义区与相邻帧的光流,以此为向导腐蚀无定义区;利用拼接的方法,填充仍为无定义区的像素。实验结果表明该算法对于前景物体运动具有较好的鲁棒性并能够生成内容完整的稳定视频序列。     

Human-vision-based real-time stereopsis

The progression in the field of stereoscopic imaging has resulted in impressive 3D videos. This technology is now used for commercial and entertainment purposes and sometimes even for medical applications. Currently, it is impossible to produce quality anaglyph video using a single camera under different moving and atmospheric conditions with the corresponding depth, local colour, and structural information. The proposed study challenges the previous researches by introducing single camera based method for anaglyph reconstruction and it mainly concentrates on human visual perception, where as the previous methods used dual camera, depth sensor, multi view, which demand not only long duration they also suffer from photometric distortion due to variation in angular alignment. This study also contributes clear individual image without any occlusion with another image. We use an approach based on human vision to determine the corresponding depth information. The source frames are shifted slightly in opposite directions as the distance between the pupils increases. We integrate the colour components of the shifted frames to generate contrasting colours for each one of the marginally shifted frames. The colour component images are then reconstructed as a cyclopean image. We show the results of our method by applying it to quickly varying video sequences and compare its performance to other existing methods.     

Video dehazing with spatial and temporal coherence

This paper describes a new framework for video dehazing, the process of restoring the visibility of the videos taken under foggy scenes. The framework builds upon techniques in single image dehazing, optical flow estimation and Markov random field. It aims at improving the temporal and spatial coherence of the dehazed video. In this framework, we first extract the transmission map frame-by-frame using guided filter, then estimate the forward and backward optical flow between two neighboring frames to find the matched pixels. The flow fields are used to help us building an MRF model on the transmission map to improve the spatial and temporal coherence of the transmission. The proposed algorithm is verified in both real and synthetic videos. The results demonstrate that our algorithm can preserve the spatial and temporal coherence well. With more coherent transmission map, we get better refocusing effect. We also apply our framework on improving the video coherence on the application of video denoising.     

Unsupervised Video Hashing via Deep Neural Network

Hashing is a common solution for content-based multimedia retrieval by encoding high-dimensional feature vectors into short binary codes. Previous works mainly focus on image hashing problem. However, these methods can not be directly used for video hashing, as videos contain not only spatial structure within each frame, but also temporal correlation between successive frames. Several researchers proposed to handle this by encoding the extracted key frames, but these frame-based methods are time-consuming in real applications. Other researchers proposed to characterize the video by averaging the spatial features of frames and then the existing hashing methods can be adopted. Unfortunately, the sort of “video” features does not take the correlation between frames into consideration and may lead to the loss of the temporal information. Therefore, in this paper, we propose a novel unsupervised video hashing framework via deep neural network, which performs video hashing by incorporating the temporal structure as well as the conventional spatial structure. Specially, the spatial features of videos are obtained by utilizing convolutional neural network, and the temporal features are established via long-short term memory. After that, the time series pooling strategy is employed to obtain the single feature vector for each video. The obtained spatio-temporal feature can be applied to many existing unsupervised hashing methods. Experimental results on two real datasets indicate that by employing the spatio-temporal features, our hashing method significantly improves the performance of existing methods which only deploy the spatial features, and meanwhile obtains higher mean average precision compared with the state-of-the-art video hashing methods.     

Video summarization via exploring the global and local importance

Video Summarization is to generate an important or interesting short video from a long video. It is important to reduce the time required to analyze the same archived video by removing unnecessary video data. This work proposes a novel method to generate dynamic video summarization by fusing the global importance and local importance based on multiple features and image quality. First, videos are split into several suitable video clips. Second, video frames are extracted from each video clip, and the center parts of frames are also extracted. Third, for each frame and the center part, the global importance and the local importance are calculated by using a set of features and image quality. Finally, the global importance and the local importance are fused to select an optimal subset for generating video summarization. Extensive experiments are conducted to demonstrate that the proposed method enables to generate high-quality video summarization.     

三焦点张量重投影视频稳像算法

目的 目前,特征点轨迹稳像算法无法兼顾轨迹长度、鲁棒性及轨迹利用率,因此容易造成该类算法的视频稳像结果扭曲失真或者局部不稳。针对此问题,提出基于三焦点张量重投影的特征点轨迹稳像算法。方法 利用三焦点张量构建长虚拟轨迹,通过平滑虚拟轨迹定义稳定视图,然后利用三焦点张量将实特征点重投影到稳定视图,以此实现实特征点轨迹的平滑,最后利用网格变形生成稳定帧。结果 对大量不同类型的视频进行稳像效果测试,并且与典型的特征点轨迹稳像算法以及商业软件进行稳像效果对比,其中包括基于轨迹增长的稳像算法、基于对极几何点转移的稳像算法以及商业软件Warp Stabilizer。本文算法的轨迹长度要求低、轨迹利用率高以及鲁棒性好,对于92%剧烈抖动的视频,稳像效果优于基于轨迹增长的稳像算法;对于93%缺乏长轨迹的视频以及71.4%存在滚动快门失真的视频,稳像效果优于Warp Stabilizer;而与基于对极几何点转移的稳像算法相比,退化情况更少,可避免摄像机阶段性静止、摄像机纯旋转等情况带来的算法失效问题。结论 本文算法对摄像机运动模式和场景深度限制少,不仅适宜处理缺少视差、场景结构非平面、滚动快门失真等常见的视频稳像问题,而且在摄像机摇头、运动模糊、剧烈抖动等长轨迹缺乏的情况下,依然能取得较好的稳像效果,但该算法的时间性能还有所不足。     

基于递归对齐网络的黑白老卡通高清重制

黑白老卡通视频在数字化的过程中会出现诸如划痕、脏点、模糊和分辨率过低等复合问题,老卡通视频增强是视频增强的一类特殊子问题,当前尚缺乏针对性算法,因此提出一种多帧联合的递归对齐增强网络解决老卡通中的复合问题。首先通过递归结构传递重建历史中的长时域信息对划痕与脏点进行修复,解决了连续性划痕与脏点的处理难题。然后在递归单元中通过基于可变形卷积的对齐模块进行相邻帧特征对齐,改善了网络在卡通大幅度运动场景中提取和补充帧间细节的能力。在递归单元末端设计了级联金字塔结构的多尺度重建模块促进特征聚合,以充分挖掘重建所需的时间和空间细节信息。实验使用峰值信噪比等客观评估标准,在降质数据集和真实老卡通数据集上进行实验测试,并与其他主流方法进行对比。实验结果表明,该方法相比于其他主流视频增强方法有较为明显提升,同时在真实黑白老卡通上可获取高视觉质量的重建结果。     

A two-step approach to see-through bad weather for surveillance video quality enhancement

Adverse weather conditions such as snow, fog or heavy rain greatly reduce the visual quality of outdoor surveillance videos. Video quality enhancement can improve the visual quality of surveillance videos providing clearer images with more details to better meet human perception needs and also improve video analytics performance. Existing work in this area mainly focuses on the quality enhancement for high-resolution videos or still images, but few algorithms are developed for enhancing surveillance videos, which normally have low resolution, high noises and compression artifacts. In addition, for snow or rain conditions, the image quality of near-field view is degraded by the obscuration of apparent snowflakes or raindrops, while the quality of far-field view is degraded by the obscuration of fog-like snowflakes or raindrops. Very few video quality enhancement algorithms have been developed to handle both problems. In this paper, we propose a novel video quality enhancement algorithm for see-through snow, fog or heavy rain. Our algorithm not only improves human visual perception experiences for video surveillance, but also reveal more video contents for better video content analyses. The proposed algorithm handles both near-field and far-field snow/rain effects by proposed a two-step approach: (1) the near-field enhancement algorithm identifies obscuration pixels by snow or rain in the near-field view and removes these pixels as snowflakes or raindrops; different from state-of-the-art methods, our proposed algorithm in this step can detect snowflakes on foreground objects or background, and apply different methods to fill in the removed regions. (2) The far-field enhancement algorithm restores the image’s contrast information not only to reveal more details in the far-field view, but also to enhance the overall image’s quality; in this step, the proposed algorithm adaptively enhances the global and local contrast, which is inspired on the human visual system, and accounts for the perceptual sensitivity to noises, compression artifacts, and the texture of image content. From our extensive testing, the proposed approach significantly improves the visual quality of surveillance videos by removing snow/fog/rain effects.