Robust optical flow estimation via edge preserving filtering

It is known that optical flow estimation techniques suffer from the issues of ill-defined edges and boundaries of the moving objects. Traditional variational methods for optical flow estimation are not robust to handle these issues since the local filters in these methods do not hold the robustness near the edges. In this paper, we propose a non-local total variation NLTV-$L^1$ optical flow estimation method based on robust weighted guided filtering. Specifically, first, the robust weighted guided filtering objective function is proposed to preserve motion edges. The proposed objective function is based on the linear model which is computationally efficient and edge-preserving in complex natural scenarios. Second, the proposed weighted guided filtering objective function is incorporated into the non-local total variation NLTV-$L^1$ energy function. Finally, the novel NLTV-$L^1$ optical flow method is performed using the coarse-to-fine process. Additionally, we modify some state-of-the-art variational optical flow estimation methods by the robust weighted guided filtering objective function to verify the performance on Middlebury, MPI-Sintel, and Foggy Zurich sequences. Experimental results show that the proposed method can preserve edges and improve the accuracy of optical flow estimation compared with several state-of-the-art methods.     

非刚性稠密匹配大位移运动光流估计

光流场是目标检测,无人机定位等众多计算机视觉任务的重要基础.本文针对非刚性大位移运动等困难运动类型图像序列光流计算的准确性与鲁棒性问题,提出一种基于非刚性稠密匹配的TV-L1（Total Variational with L1 norm,TV-L1）大位移光流计算方法.首先,使用非刚性稠密块匹配计算图像序列初始最近邻域场,其次根据图像相邻块区域的相似性消除初始最近邻域场中的非一致性区域以得到准确的图像最近邻域场.然后,在图像金字塔分层计算框架下,将图像最近邻域场引入基于非局部约束的TV-L1光流估计模型,通过Quadratic Pseudo-Boolean Optimization（QPBO）融合算法在金字塔分层图像光流计算时对TV-L1模型光流估计进行大位移运动补偿.最后,采用标准测试图像序列对本文方法和当前代表性的变分方法LDOF（Large Displacement Optical Flow,LDOF）、Classic+NL、NNF（Nearest Neighbor Fields,NNF）以及深度学习方法FlowNet2.0进行对比分析.实验结果表明,本文方法能有效提高非刚性运动、大位移运动以及运动遮挡等困难运动类型光流估计的精度与鲁棒性.     

基于图像局部结构的区域匹配变分光流算法

针对变分光流算法的计算精度与鲁棒性问题,提出一种基于图像局部结构的区域匹配变分光流算法.光流估计能量泛函的数据项采用图像结构守恒与灰度守恒相结合,并引入规则化非平方惩罚函数,保证了光流估计的精度与鲁棒性;平滑项采用随图像局部结构自适应变化的扩散策略结合区域匹配约束函数能够有效地保护运动物体或场景的边缘轮廓信息;在光流计算过程中引入金字塔分层细化策略克服图像序列中大位移运动引起的像素点漂移现象,并采用数学方法证明光流估计模型的鲁棒性和收敛性.多组实验表明,本文方法在图像中存在剧烈光照变化、非刚性物体复杂运动以及多目标大位移运动等情况下具有较高的计算精度、较好的鲁棒性.     

应用于稳像系统中的改进梯度光流法

由于传统的梯度光流法当运动不连续时运动场估计值与真实值有较大偏差,因而不能直接应用于稳像系统中。引入金字塔多分辨率分层技术对传统的梯度光流法进行改进。首先,在视频序列中选定细节丰富的区域作为计算区域;其次,利用结合金字塔多分辨率分层技术的光流法迭代求解相邻帧间的仿射参数;最后,采用帧间补偿方式,并增加控制累积错误传播的措施,在不丢失过多原有信息基础上,实现了长时间稳像。实验表明:改进的方法能够检测出剧烈的复杂抖动,并能达到旋转精度小于0.09、平移精度小于0.07个像素,缩放精度小于0.02的高精度估计,且补偿序列平均峰值信噪比值提高了2.36 dB以上。     

Optical flow estimation based on the structure–texture image decomposition

Optical flow approaches for motion estimation calculate vector fields which determine the apparent velocities of objects in time-varying image sequences. Image motion estimation is a fundamental issue in low-level vision and is used in many applications in image sequence processing, such as robot navigation, object tracking, image coding and structure reconstruction. The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. Actually, several methods are used to estimate the optical flow, but a good compromise between computational cost and accuracy is hard to achieve. This work presents a combined local–global total variation approach with structure–texture image decomposition. The combination is used to control the propagation phenomena and to gain robustness against illumination changes, influence of noise on the results and sensitivity to outliers. The resulted method is able to compute larger displacements in a reasonable time.     

基于自相似性约束的视频稀疏超分辨率重建

通过研究帧间自相似性对图像重建的影响,提出一种自相似性约束的单视频稀疏超分辨率重建算法,以达到保持图像局部结构完整性的同时有效去噪的目的。该算法运用主成分分析PCA训练出适应图像不同局部结构的分类词典;通过帧间光流场的粗略运动估计和帧内帧间的精确块匹配,搜索自相似信息,运用非局部均值NLM滤波,并以此约束稀疏模型。仿真实验表明,提出的算法无论是客观指标,还是主观视觉上都超过了进行比较的几种分辨率提高算法。     

Video Super-resolution using Edge-based Optical Flow and Intensity Prediction

Full-image based motion prediction is widely used in video super-resolution (VSR) that results outstanding outputs with arbitrary scenes but costs huge time complexity. In this paper, we propose an edge-based motion and intensity prediction scheme to reduce the computation cost while maintain good enough quality simultaneously. The key point of reducing computation cost is to focus on extracted edges rather than the whole frame when finding motion vectors (optical flow) of the video sequence in accordance with human vision system (HVS). Bi-directional optical flow is usually adopted to increase the prediction accuracy but it also increase the computation time. Here we propose to obtain the backward flow from foregoing forward flow prediction which effectively save the heavy load. We perform a series of experiments and comparisons between existed VSR methods and our proposed edge-based method with different sequences and upscaling factors. The results reveal that our proposed scheme can successfully keep the super-resolved sequence quality and get about 4x speed up in computation time.     

相互结构引导滤波TV-L1变分光流估计

由于光流场既包含物体的运动信息,又包含场景的三维结构信息,因此光流计算技术是计算机视觉和机器视觉领域研究的重要任务之一.针对现有光流计算方法在图像边缘保护方面存在过度平滑问题,提出一种基于相互结构引导滤波的TV-L1（Total Variational with L1 norm,TV-L1）变分光流估计方法.通过提取置信度较高的图像相互结构区域,构造基于相互结构引导滤波的全局目标函数,并采用金字塔分层细化与交替迭代方案结合的策略进行优化,该方法可以较好的保护图像边缘信息.最后采用标准测试图像集对本文方法与现有代表性变分方法LDOF（Large Displacement Optical Flow,LDOF）,CLG-TV（Combined Local-Global Total Variation,CLG-TV）,Classic++,NNF（Nearest Neighbor Fields,NNF）以及深度学习方法FlowNet2.0进行对比,实验结果表明本文方法具有较高的光流估计精度与鲁棒性,尤其对图像边缘保护具有显著的效果,并且在运动目标检测,机器人避障等方面具有一定应用前景.     

Optical flow computation using extended constraints

Several approaches for optical flow estimation use partial differential equations to model changes in image brightness throughout time. A commonly used equation is the so-called optical flow constraint (OFC), which assumes that the image brightness is stationary with respect to time. More recently, a different constraint referred to as the extended optical flow constraint (EOFC) has been introduced, which also contains the divergence of the flow field of image brightness. There is no agreement in the literature about which of these constraints provides the best estimation of the velocity field. Two new solutions for optical flow computation are proposed, which are based on an approximation of the constraint equations. The two techniques have been used with both EOFC and OFC constraint equations. Results achieved by using these solutions have been compared with several well-known computational methods for optical flow estimation in different motion conditions. Estimation errors have also been measured and compared for different types of motion.     

Accurate optical flow in noisy image sequences using flow adapted anisotropic diffusion

In this paper, we combine 3D anisotropic diffusion and motion estimation for image denoising and improvement of motion estimation. We compare different continuous isotropic nonlinear and anisotropic diffusion processes, which can be found in literature, with a process especially designed for image sequence denoising for motion estimation. All of these processes initially improve motion estimation due to reduction of noise and high frequencies. But while all the well known processes rapidly destroy or hallucinate motion information, the process brought forward here shows considerably less information loss or violation even at motion boundaries. We show the superior behavior of this process. Further we compare the performance of a standard finite difference diffusion scheme with several schemes using derivative filters optimized for rotation invariance. Using the discrete scheme with least smoothing artifacts we demonstrate the denoising capabilities of this approach. We exploit the motion estimation to derive an automatic stopping criterion.     

Regularization of optic flow estimates by means of weighted vectormedian filtering

Vector median filtering has been recently proposed as an effective method to refine estimated velocity fields. Here, the use of a weighted vector median filtering is suggested to improve the regularization of the optic flow field across motion boundaries. Information about the confidence of the estimated pixel velocities is exploited for the choice of the filter weights. Experimental results, on both synthetic and real-world sequences, show the effectiveness of the proposed procedure.     

基于语义分割的双目场景流估计

针对现有场景流计算方法在复杂场景、大位移和运动遮挡等情况下易产生运动边缘模糊的问题,提出一种基于语义分割的双目场景流估计方法.首先,根据图像中的语义信息类别,通过深度学习的卷积神经网络模型将图像划分为带有语义标签的区域;针对不同语义类别的图像区域分别进行运动建模,利用语义知识计算光流信息并通过双目立体匹配的半全局匹配方法计算图像视差信息.然后,对输入图像进行超像素分割,通过最小二乘法耦合光流和视差信息,分别求解每个超像素块的运动参数.最后,在优化能量函数中添加语义分割边界的约束信息,通过更新像素到超像素块的映射关系和超像素块到移动平面的映射关系得到最终的场景流估计结果.采用KITTI 2015标准测试图像序列对本文方法和代表性的场景流计算方法进行对比分析.实验结果表明,本文方法具有较高的精度和鲁棒性,尤其对于复杂场景、运动遮挡和运动边缘模糊的图像具有较好的边缘保护作用.     

Optical flow estimation and moving object segmentation based onmedian radial basis function network

Various approaches have been proposed for simultaneous optical flow estimation and segmentation in image sequences. In this study, the moving scene is decomposed into different regions with respect to their motion, by means of a pattern recognition scheme. The inputs of the proposed scheme are the feature vectors representing still image and motion information. Each class corresponds to a moving object. The classifier employed is the median radial basis function (MRBF) neural network. An error criterion function derived from the probability estimation theory and expressed as a function of the moving scene model is used as the cost function. Each basis function is activated by a certain image region. Marginal median and median of the absolute deviations from the median (MAD) estimators are employed for estimating the basis function parameters. The image regions associated with the basis functions are merged by the output units in order to identify moving objects.     

DCT域快速下采样运动向量滤波器

目前的视频压缩标准多数采用DCT变换编码和运动补偿技术。运动估计约占整个编码时间的60％、运动补偿约占10％。所以在视频转码中，运动向量的再使用技术是十分重要的，目前较好的方法是欧氏最小距离方法，它的主要缺点是估计精度不高，本文对此进行了改进，提出了DCT、域快速下采样运动向量滤波器，其重建图像的峰值信噪声比Shanableh等人提出的方法平均高0．2dB。     

稳健的预分析多直线全局运动估计方法

本文提出了RTSG图像预分析和多直线全局运动估计方法,该方法首先根据光流场方程得到RTSG图像,并对RTSG图像进行预分析以去除图像中的外点和确定全局运动模型,然后提取多个直线特征元,利用直线特征元估计局部运动参数,最后用局部运动参数和与之相关的权值进行全局运动估计。     

Variational method for super-resolution optical flow

The motion fields in an image sequence observed by a car-mounted imaging system depend on the positions in the imaging plane. Since the motion displacements in the regions close to the camera centre are small, for accurate optical flow computation in this region, we are required to use super-resolution of optical flow fields. We develop an algorithm for super-resolution optical flow computation. Super-resolution of images is a technique for recovering a high-resolution image from a low-resolution image and/or image sequence. Optical flow is the appearance motion of points on the image. Therefore, super-resolution optical flow computation yields the appearance motion of each point on the high-resolution image from a sequence of low-resolution images. We combine variational super-resolution and variational optical flow computation in super-resolution optical flow computation. Our method directly computes the gradient and spatial difference of high-resolution images from those of low-resolution images, without computing any high-resolution images used as intermediate data for the computation of optical flow vectors of the high-resolution image.     

单目图像序列光流三维重建技术研究综述

由单目图像序列光流重建物体或场景的三维运动与结构是计算机视觉、图像处理与模式识别等领域的重要研究内容,在机器人视觉、无人机导航、车辆辅助驾驶以及医学影像分析等方面具有重要的应用。本文首先从精度与鲁棒性等方面对单目图像序列光流计算及三维重建技术近年来取得的进展进行综述与分析。然后采用Middlebury测试图像序列对HS、LDOF、CLG-TV、SOF、AOFSCNN 和 Classic ＋NL 等典型光流算法以及 Adiv、RMROF、Sekkati 和DMDPOF等基于光流的间接与直接重建方法进行实验对比分析,指出各对比方法的优点与不足,归纳各类方法的性能特点与适用范围。最后对利用分数阶微分模型、非局部约束、立体视觉以及深度线索解决亮度突变、非刚性运动、运动遮挡与模糊情况下光流计算及重建模型的局限性与鲁棒性问题进行总结与展望。     

基于双边滤波与非局部均值的图像去噪研究

提出了一种基于双边滤波与非局部均值的图像去 噪算法,近年来提出的非局部均值算法是去噪效果非常 出色的算法之一,双边滤波去噪算法采用空间邻近度和灰度相似性构造新的权重系数,其取 得了良好的去噪效果,本文 据此改进非局部均值算法的权重部分,把空间邻近度因子与非局部均值的权重系数相结合, 构造新的权重系数。实验表 明,本文改进权重的非局部均值算法与已有的去噪方法相比,能得到更好的峰值信噪比,能 更好的保护图像细节以及结构信息。     

Image denoising based on non-local means filter and its method noise thresholding

Non-local means filter uses all the possible self-predictions and self-similarities the image can provide to determine the pixel weights for filtering the noisy image, with the assumption that the image contains an extensive amount of self-similarity. As the pixels are highly correlated and the noise is typically independently and identically distributed, averaging of these pixels results in noise suppression thereby yielding a pixel that is similar to its original value. The non-local means filter removes the noise and cleans the edges without losing too many fine structure and details. But as the noise increases, the performance of non-local means filter deteriorates and the denoised image suffers from blurring and loss of image details. This is because the similar local patches used to find the pixel weights contains noisy pixels. In this paper, the blend of non-local means filter and its method noise thresholding using wavelets is proposed for better image denoising. The performance of the proposed method is compared with wavelet thresholding, bilateral filter, non-local means filter and multi-resolution bilateral filter. It is found that performance of proposed method is superior to wavelet thresholding, bilateral filter and non-local means filter and superior/akin to multi-resolution bilateral filter in terms of method noise, visual quality, PSNR and Image Quality Index.