基于深度匹配的由稀疏到稠密大位移运动光流估计

针对非刚性大位移运动场景的光流计算准确性与鲁棒性问题, 提出一种基于深度匹配的由稀疏到稠密大位移运动光流估计方法. 首先利用深度匹配模型计算图像序列相邻帧的初始稀疏运动场; 其次采用网格化邻域支持优化模型筛选具有较高置信度的图像网格和匹配像素点, 获得鲁棒的稀疏运动场; 然后对稀疏运动场进行边缘保护稠密插值, 并设计全局能量泛函优化求解稠密光流场. 最后分别利用MPI-Sintel和KITTI数据库提供的测试图像集对本文方法和Classic + NL, DeepFlow, EpicFlow以及FlowNetS等变分模型、匹配策略和深度学习光流计算方法进行综合对比与分析, 实验结果表明本文方法相对于其他方法具有更高的光流计算精度, 尤其在非刚性大位移和运动遮挡区域具有更好的鲁棒性与可靠性.     

FALDOI: A New Minimization Strategy for Large Displacement Variational Optical Flow

We propose a large displacement optical flow method that introduces a new strategy to compute a good local minimum of any optical flow energy functional. The method requires a given set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from those matches. In particular, the matches are used to guide a structured coordinate descent of the energy functional around these keypoints. It results in a two-step minimization method at the finest scale which is very robust to the inevitable outliers of the sparse matcher and able to capture large displacements of small objects. Its benefits over other variational methods that also rely on a set of sparse matches are its robustness against very few matches, high levels of noise, and outliers. We validate our proposal using several optical flow variational models. The results consistently outperform the coarse-to-fine approaches and achieve good qualitative and quantitative performance on the standard optical flow benchmarks.     

A Multigrid Platform for Real-Time Motion Computation with Discontinuity-Preserving Variational Methods

Variational methods are among the most accurate techniques for estimating the optic flow. They yield dense flow fields and can be designed such that they preserve discontinuities, estimate large displacements correctly and perform well under noise and varying illumination. However, such adaptations render the minimisation of the underlying energy functional very expensive in terms of computational costs: Typically one or more large linear or nonlinear equation systems have to be solved in order to obtain the desired solution. Consequently, variational methods are considered to be too slow for real-time performance. In our paper we address this problem in two ways: (i) We present a numerical framework based on bidirectional multigrid methods for accelerating a broad class of variational optic flow methods with different constancy and smoothness assumptions. Thereby, our work focuses particularly on regularisation strategies that preserve discontinuities. (ii) We show by the examples of five classical and two recent variational techniques that real-time performance is possible in all cases—even for very complex optic flow models that offer high accuracy. Experiments show that frame rates up to 63 dense flow fields per second for image sequences of size 160 × 120 can be achieved on a standard PC. Compared to classical iterative methods this constitutes a speedup of two to four orders of magnitude.     

一种基于光流场重建三维运动和结构的新方法

提出了一种基于稀疏光流场计算三维运动和结构的线性新方法 ,该方法综合视觉运动分析中的两类处理方法 ,选取图象中的角点作为特征点 ;并检测和跟踪图象序列中的角点 .记录检测到的角点在图象序列中的位移 ,在理论上证明了时变图象的光流场可以近似地用角点的位移场代替 ,从而得到时变图象的稀疏光流场 ;通过光流运动模型的建立 ,推导出由稀疏光流场重建三维物体运动和结构的线性方法 .通过用真实图象序列验证该算法 ,表明该算法取得了较好的效果     

EGGS: Sparsity-Specific Code Generation

Sparse matrix computations are among the most important computational patterns, commonly used in geometry processing, physical simulation, graph algorithms, and other situations where sparse data arises. In many cases, the structure of a sparse matrix is known a priori, but the values may change or depend on inputs to the algorithm. We propose a new methodology for compile-time specialization of algorithms relying on mixing sparse and dense linear algebra operations, using an extension to the widely-used open source Eigen package. In contrast to library approaches optimizing individual building blocks of a computation (such as sparse matrix product), we generate reusable sparsity-specific implementations for a given algorithm, utilizing vector intrinsics and reducing unnecessary scanning through matrix structures. We demonstrate the effectiveness of our technique on a benchmark of artificial expressions to quantitatively evaluate the benefit of our approach over the state-of-the-art library Intel MKL. To further demonstrate the practical applicability of our technique we show that our technique can improve performance, with minimal code changes, for mesh smoothing, mesh parametrization, volumetric deformation, optical flow, and computation of the Laplace operator.     

Appearance Flow Completion for Novel View Synthesis

Novel view synthesis from sparse and unstructured input views faces challenges like the difficulty with dense 3D reconstruction and large occlusion. This paper addresses these problems by estimating proper appearance flows from the target to input views to warp and blend the input views. Our method first estimates a sparse set 3D scene points using an off‐the‐shelf 3D reconstruction method and calculates sparse flows from the target to input views. Our method then performs appearance flow completion to estimate the dense flows from the corresponding sparse ones. Specifically, we design a deep fully convolutional neural network that takes sparse flows and input views as input and outputs the dense flows. Furthermore, we estimate the optical flows between input views as references to guide the estimation of dense flows between the target view and input views. Besides the dense flows, our network also estimates the masks to blend multiple warped inputs to render the target view. Experiments on the KITTI benchmark show that our method can generate high quality novel views from sparse and unstructured input views.     

A Variational Aggregation Framework for Patch-Based Optical Flow Estimation

We propose a variational aggregation method for optical flow estimation. It consists of a two-step framework, first estimating a collection of parametric motion models to generate motion candidates, and then reconstructing a global dense motion field. The aggregation step is designed as a motion reconstruction problem from spatially varying sets of motion candidates given by parametric motion models. Our method is designed to capture large displacements in a variational framework without requiring any coarse-to-fine strategy. We handle occlusion with a motion inpainting approach in the candidates computation step. By performing parametric motion estimation, we combine the robustness to noise of local parametric methods with the accuracy yielded by global regularization. We demonstrate the performance of our aggregation approach by comparing it to standard variational methods and a discrete aggregation approach on the Middlebury and MPI Sintel datasets.     

A Variational Approach to Video Registration with Subspace Constraints

This paper addresses the problem of non-rigid video registration, or the computation of optical flow from a reference frame to each of the subsequent images in a sequence, when the camera views deformable objects. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms, leading to an efficient optimization scheme. Additionally, we propose a novel optimization scheme for the case of vector valued images, based on the dualization of the data term. This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results. Finally, we provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evaluation of multi-frame optical flow algorithms for non-rigid surfaces. Our experiments show that our proposed approach outperforms state of the art optical flow and dense non-rigid registration algorithms.     

A loop-consistency measure for dense correspondences in multi-view video

Many applications in computer vision and computer graphics require dense correspondences between images of multi-view video streams. Most state-of-the-art algorithms estimate correspondences by considering pairs of images. However, in multi-view videos, several images capture nearly the same scene. In this article we show that this redundancy can be exploited to estimate more robust and consistent correspondence fields. We use the multi-video data structure to establish a confidence measure based on the consistency of the correspondences in a loop of three images. This confidence measure can be applied after flow estimation is terminated to find the pixels for which the estimate is reliable. However, including the measure directly into the estimation process yields dense and highly accurate correspondence fields. Additionally, application of the loop consistency confidence measure allows us to include sparse feature matches directly into the dense optical flow estimation. With the confidence measure, spurious matches can be successfully suppressed during optical flow estimation while correct matches contribute to increase the accuracy of the flow.     

Differential optical flow applied to automatic facial expression recognition

This work compares systematically two optical flow-based facial expression recognition methods. The first one is featural and selects a reduced set of highly discriminant facial points while the second one is holistic and uses much more points that are uniformly distributed on the central face region. Both approaches are referred as feature point tracking and holistic face dense flow tracking, respectively. They compute the displacements of different sets of points along the sequence of frames describing each facial expression (i.e. from neutral to apex). First, we evaluate our algorithms on the Cohn-Kanade database for the six prototypic expressions under two different spatial frame resolutions (original and 40%-reduced). Later, our methods were also tested on the MMI database which presents higher variabilities than the Cohn-Kanade one. The results on the first database show that dense flow tracking method at original resolution slightly outperformed, in average, the recognition rates of feature point tracking method (95.45% against 92.42%) but it requires 68.24% more time to track the points. For the patterns of MMI database, using dense flow tracking at the original resolution, we achieved very similar average success rates.     

Multi-view structure-from-motion for hybrid camera scenarios

We describe a pipeline for structure-from-motion (SfM) with mixed camera types, namely omnidirectional and perspective cameras. For the steps of this pipeline, we propose new approaches or adapt the existing perspective camera methods to make the pipeline effective and automatic. We model our cameras of different types with the sphere camera model. To match feature points, we describe a preprocessing algorithm which significantly increases scale invariant feature transform (SIFT) matching performance for hybrid image pairs. With this approach, automatic point matching between omnidirectional and perspective images is achieved. We robustly estimate the hybrid fundamental matrix with the obtained point correspondences. We introduce the normalization matrices for lifted coordinates so that normalization and denormalization can be performed linearly for omnidirectional images. We evaluate the alternatives of estimating camera poses in hybrid pairs. A weighting strategy is proposed for iterative linear triangulation which improves the structure estimation accuracy. Following the addition of multiple perspective and omnidirectional images to the structure, we perform sparse bundle adjustment on the estimated structure by adapting it to use the sphere camera model. Demonstrations of the end-to-end multi-view SfM pipeline with the real images of mixed camera types are presented.     

基于角点跟踪的光流场计算

在综合视觉运动分析中的两类处理方法，选取图像中的角点作为特征点，并检测和跟踪图像序列中的角点。记录检测到的角点在图像序列中的位移，在理论上证明了时变图像的光流场可以近似地用角点的位移场代替，同时给出这种替代的两个前提条件。本文用真实图像序列验证提出的算法，实验结果表明该算法取得了较好的效果。     

基于低秩稀疏评分的非监督特征选择

在处理高维数据过程中,特征选择是一个非常重要的数据降维步骤。低秩表示模型具有揭示数据全局结构信息的能力和一定的鉴别能力。稀疏表示模型能够利用较少的连接关系揭示数据的本质结构信息。在低秩表示模型的基础上引入稀疏约束项,构建一种低秩稀疏表示模型学习数据间的低秩稀疏相似度矩阵;基于该矩阵提出一种低秩稀疏评分机制用于非监督特征选择。在不同数据库上将选择后的特征进行聚类和分类实验,同传统特征选择算法进行比较。实验结果表明了低秩特征选择算法的有效性。     

一种局部和全局相结合的光流计算方法

光流场是计算机视觉的一个研究方向,微分法是计算光流场的一个常用方法,它分为全局方法和局部方法,全局方法能够得到100％的致密的光流场,而局部方法大多只能得到稀疏的光流场,但它在噪声情况下具有更好的鲁棒性。本文提出一种局部和全局相结合的方法．首先给出五点光流约束的局部方法,再结合全局方法,计算得到了既致密又 鲁棒的光流场。     

基于纹理约束和参数化运动模型的光流估计

提出了一种基于局部小平面运动的光流估计新方法。目的是获得精确致密的光流估计结果。与以往采用亮度一致性区域作为假设平面的算法不同，本算法利用序列图像的纹理信息，在纹理分割区域的基础上，进行运动估计。该算法首先通过微分法计算粗光流，可以得到参数化光流模型的初始估计，然后通过区域迭代算法，调整初始估计，从而得到精细的平面分割及其对应的参数化光流模型。基于纹理信息的部分拟合算法被用于算法的每一步当中，保证了纹理边缘位置的光流估计值的准确性。实验采用了标准图像序列，结果表明，可以得到更为精细的光流估计结果，特别是对于那些有着丰富纹理信息的室外环境的图像序列，而且在运动边界处的结果改善尤为明显。     

Multi-objective optimization for parameter selection and characterization of optical flow methods

Optical flow methods are among the most accurate techniques for estimating displacement and velocity fields in a number of applications that range from neuroscience to robotics. The performance of any optical flow method will naturally depend on the configuration of its parameters, and for different applications there are different trade-offs between the corresponding evaluation criteria (e.g. the accuracy and the processing speed of the estimated optical flow). Beyond the standard practice of manual selection of parameters for a specific application, in this article we propose a framework for automatic parameter setting that allows searching for an approximated Pareto-optimal set of configurations in the whole parameter space. This final Pareto-front characterizes each specific method, enabling proper method comparison and proper parameter selection. Using the proposed methodology and two open benchmark databases, we study two recent variational optical flow methods. The obtained results clearly indicate that the method to be selected is application dependent, that in general method comparison and parameter selection should not be done using a single evaluation measure, and that the proposed approach allows to successfully perform the desired method comparison and parameter selection.     

Sparse representation for robust abnormality detection in crowded scenes

In crowded scenes, the extracted low-level features, such as optical flow or spatio-temporal interest point, are inevitably noisy and uncertainty. In this paper, we propose a fully unsupervised non-negative sparse coding based approach for abnormality event detection in crowded scenes, which is specifically tailored to cope with feature noisy and uncertainty. The abnormality of query sample is decided by the sparse reconstruction cost from an atomically learned event dictionary, which forms a sparse coding bases. In our algorithm, we formulate the task of dictionary learning as a non-negative matrix factorization (NMF) problem with a sparsity constraint. We take the robust Earth Mover's Distance (EMD), instead of traditional Euclidean distance, as distance metric reconstruction cost function. To reduce the computation complexity of EMD, an approximate EMD, namely wavelet EMD, is introduced and well combined into our approach, without losing performance. In addition, the combination of wavelet EMD with our approach guarantees the convexity of optimization in dictionary learning. To handle both local abnormality detection (LAD) and global abnormality detection, we adopt two different types of spatio-temporal basis. Experiments conducted on four public available datasets demonstrate the promising performance of our work against the state-of-the-art methods.     

Video synchronization and its application to object transfer

In video post-production applications, camera motion analysis and alignment are important in order to ensure the geometric correctness and temporal consistency. In this paper, we trade some generality in estimating and aligning camera motion for reduced computational complexity and increased image-based nature. The main contribution is to use fundamental ratios to synchronize video sequences of distinct scenes captured by cameras undergoing similar motions. We also present a simple method to align 3D camera trajectories when the fundamental ratios are not able to match the noisy trajectories. Experimental results show that our method can accurately synchronize sequences even when the scenes are totally different and have dense depths. An application on 3D object transfer is also demonstrated.     

Joint segmentation of collectively moving objects using a bag-of-words model and level set evolution

In scenes with collectively moving objects, to disregard the individual objects and take the entire group into consideration for motion characterization is a promising approach with wide application prospects. In contrast to studies on the segmentation of independently moving objects, our purpose is to construct a segmentation of these objects to characterize their motions at a macroscopic level. In general, the collectively moving objects in a group have very similar motion behavior with their neighbors and appear as a kind of global collective motion. This paper presents a joint segmentation approach for these collectively moving objects. In our model, we extract these macroscopic movement patterns based on optical flow field sequences. Specifically, a group of collectively moving objects correspond to a region where the optical flow field has high magnitude and high local direction coherence. As a result, our problem can be addressed by identifying these coherent optical flow field regions. The segmentation is performed through the minimization of a variational energy functional derived from the Bayes classification rule. Specifically, we use a bag-of-words model to generate a codebook as a collection of prototypical optical flow patterns, and the class-conditional probability density functions for different regions are determined based on these patterns. Finally, the minimization of our proposed energy functional results in the gradient descent evolution of segmentation boundaries which are implicitly represented through level sets. The application of our proposed approach is to segment and track multiple groups of collectively moving objects in a large variety of real-world scenes.     

Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation

Event cameras or neuromorphic cameras mimic the human perception system as they measure the per-pixel intensity change rather than the actual intensity level. In contrast to traditional cameras, such cameras capture new information about the scene at MHz frequency in the form of sparse events. The high temporal resolution comes at the cost of losing the familiar per-pixel intensity information. In this work we propose a variational model that accurately models the behaviour of event cameras, enabling reconstruction of intensity images with arbitrary frame rate in real-time. Our method is formulated on a per-event-basis, where we explicitly incorporate information about the asynchronous nature of events via an event manifold induced by the relative timestamps of events. In our experiments we verify that solving the variational model on the manifold produces high-quality images without explicitly estimating optical flow. This paper is an extended version of our previous work (Reinbacher et al. in British machine vision conference (BMVC), 2016) and contains additional details of the variational model, an investigation of different data terms and a quantitative evaluation of our method against competing methods as well as synthetic ground-truth data.