Occlusion-aware optical flow estimation

Optical flow can be reliably estimated between areas visible in two images, but not in occlusion areas. If optical flow is needed in the whole image domain, one approach is to use additional views of the same scene. If such views are unavailable, an often-used alternative is to extrapolate optical flow in occlusion areas. Since the location of such areas is usually unknown prior to optical flow estimation, this is usually performed in three steps. First, occlusion-ignorant optical flow is estimated, then occlusion areas are identified using the estimated (unreliable) optical flow, and, finally, the optical flow is corrected using the computed occlusion areas. This approach, however, does not permit interaction between optical flow and occlusion estimates. In this paper, we permit such interaction by proposing a variational formulation that jointly computes optical flow, implicitly detects occlusions and extrapolates optical flow in occlusion areas. The extrapolation mechanism is based on anisotropic diffusion and uses the underlying image gradient to preserve structure, such as optical flow discontinuities. Our results show significant improvements in the computed optical flow fields over other approaches, both qualitatively and quantitatively.     

Errors-in-variables modeling in optical flow estimation

Gradient-based optical flow estimation methods typically do not take into account errors in the spatial derivative estimates. The presence of these errors causes an errors-in-variables (EIV) problem. Moreover, the use of finite difference methods to calculate these derivatives ensures that the errors are strongly correlated between pixels. Total least squares (TLS) has often been used to address this EIV problem. However, its application in this context is flawed as TLS implicitly assumes that the errors between neighborhood pixels are independent. In this paper, a new optical flow estimation method (EIVM) is formulated to properly treat the EIV problem in optical flow. EIVM is based on Sprent's (1966) procedure which allows the incorporation of a general EIV model in the estimation process. In EIVM, the neighborhood size acts as a smoothing parameter. Due to the weights in the EIVM objective function, the effect of changing the neighborhood size is more complex than in other local model methods such as Lucas and Kanade (1981). These weights, which are functions of the flow estimate, can alter the effective size and orientation of the neighborhood. In this paper, we also present a data-driven method for choosing the neighborhood size based on Stein's unbiased risk estimators (SURE).     

STC-Flow: Spatio-temporal context-aware optical flow estimation

In this paper, we propose a spatio-temporal contextual network, STC-Flow, for optical flow estimation. Unlike previous optical flow estimation approaches with local pyramid feature extraction and multi-level correlation, we propose a contextual relation exploration architecture by capturing rich long-range dependencies in spatial and temporal dimensions. Specifically, STC-Flow contains three key context modules, i.e., pyramidal spatial context module, temporal context correlation module and recurrent residual contextual upsampling module for the effect of feature extraction, correlation, and flow reconstruction, respectively. Experimental results demonstrate that the proposed scheme achieves the state-of-the-art performance of two-frame based methods on Sintel and KITTI datasets.     

Robust optical flow estimation via edge preserving filtering

It is known that optical flow estimation techniques suffer from the issues of ill-defined edges and boundaries of the moving objects. Traditional variational methods for optical flow estimation are not robust to handle these issues since the local filters in these methods do not hold the robustness near the edges. In this paper, we propose a non-local total variation NLTV-$L^1$ optical flow estimation method based on robust weighted guided filtering. Specifically, first, the robust weighted guided filtering objective function is proposed to preserve motion edges. The proposed objective function is based on the linear model which is computationally efficient and edge-preserving in complex natural scenarios. Second, the proposed weighted guided filtering objective function is incorporated into the non-local total variation NLTV-$L^1$ energy function. Finally, the novel NLTV-$L^1$ optical flow method is performed using the coarse-to-fine process. Additionally, we modify some state-of-the-art variational optical flow estimation methods by the robust weighted guided filtering objective function to verify the performance on Middlebury, MPI-Sintel, and Foggy Zurich sequences. Experimental results show that the proposed method can preserve edges and improve the accuracy of optical flow estimation compared with several state-of-the-art methods.     

Optimal brightness functions for optical flow estimation ofdeformable motion

Estimation accuracy of Horn and Schunck's (1981) classical optical flow algorithm depends on many factors including the brightness pattern of the measured images. Since some applications can select brightness functions with which to "paint" the object, it is desirable to know what patterns will lead to the best motion estimates. The paper presents a method for determining this pattern a priori using mild assumptions about the velocity field and imaging process. The method is based on formulating Horn and Schunck's algorithm as a linear smoother and rigorously deriving an expression for the corresponding error covariance function. The authors then specify a scalar performance measure and develop an approach to select an optimal brightness function which minimizes this performance measure from within a parametrized class. Conditions for existence of an optimal brightness function are also given. The resulting optimal performance is demonstrated using simulations, and a discussion of these results and potential future research is given.     

A multi-resolution approach for massively-parallel hardware-friendly optical flow estimation

This paper presents a novel hardware-friendly motion estimation for real-time applications such as robotics or autonomous navigation. Our approach is based on the well-known Lucas & Kanade local algorithm, whose main problem is the unreliability of its estimations for large-range displacements. This disadvantage is solved in the literature by adding the sequential multiscale-with-warping extension, although it dramatically increases the computational cost. Our choice is the implementation of a multiresolution scheme that avoids the warping computation and allows the estimation of large-range motion. This alternative allows the parallel computation of the scale-by-scale motion estimation which makes the whole computation lighter and significantly reduces the processing time compared with the multiscale-with-warping approach. Furthermore, this last fact also means reducing the hardware resource cost for its potential implementation in digital hardware devices such as GPUs, ASICs, or FPGAs. In the discussion, we analyze the speedup of the multiresolution approach compared to the multiscale-with-warping scheme. For an FPGA implementation, we obtain a reduction of latency between 40% and 50% and a resource reduction of 30%. The final solution copes with large-range motion estimations with a simplified architecture very well-suited for customized digital hardware datapath implementations as well as current multicore architectures.     

采用光流估计的数字相机自动对焦算法

自动对焦技术对于数字相机至关重要,它是获取清晰图像的重要手段。针对复杂环境下多目标场景图像,提出了一种基于光流场估计的自动对焦算法。通过计算输入图像序列的光流场,对场景中的运动目标进行检测,根据目标运动属性准确判断出感兴趣目标。改进了Brenner清晰度评价方法,利用目标的二维边缘梯度信息建立评价函数,并且通过非线性增益提高评价函数的灵敏度,减小了噪声对评价值的影响。实验证明,该算法能够在主辅目标景深比达50倍的情况下分辨出感兴趣主目标,并在方差为0.02的随机噪声干扰下能有效地评价图像的清晰度;此算法将Brenner等评价函数的峰值稳定余量提高了1至4倍,对于不同图像具有良好的鲁棒性,易于硬件实现。     

B-spline image model for energy minimization-based optical flow estimation.

Robust estimation of the optical flow is addressed through a multiresolution energy minimization. It involves repeated evaluation of spatial and temporal gradients of image intensity which rely usually on bilinear interpolation and image filtering. We propose to base both computations on a single pyramidal cubic B-spline model of image intensity. We show empirically improvements in convergence speed and estimation error and validate the resulting algorithm on real test sequences.     

Robust optical flow estimation based on a sparse motion trajectory set

This paper presents an approach to the problem of estimating a dense optical flow field. The approach is based on a multiframe, irregularly spaced motion trajectory set, where each trajectory describes the motion of a given point as a function of time. From this motion trajectory set a dense flow field is estimated using a process of interpolation. A set of localized motion models are estimated, with each pixel labeled as belonging to one of the motion models. A Markov random field framework is adopted, allowing the incorporation of contextual constraints to encourage region-like structures. The approach is compared with a number of conventional optical flow estimation algorithms taken over a number of real and synthetic sequences. Results indicate that the method produces more accurate results for sequences with known ground truth flow. Also, applying the method to real sequences with unknown flow results in lower DFD, for all of the sequences tested.     

基于光流估计的整帧恢复算法

探讨了在无线环境中可能出现的整帧丢失情况，并利用光流估计的思想，提出了一种有效的双向估计恢复算法。实验结果证明，提出的改进优化算法能较好地提高恢复图像性能，并具有一定的实用价值。     

Accurate dense optical flow estimation using adaptive structure tensors and a parametric model

An accurate optical flow estimation algorithm is proposed in this paper. By combining the three-dimensional (3D) structure tensor with a parametric flow model, the optical flow estimation problem is converted to a generalized eigenvalue problem. The optical flow can be accurately estimated from the generalized eigenvectors. The confidence measure derived from the generalized eigenvalues is used to adaptively adjust the coherent motion region to further improve the accuracy. Experiments using both synthetic sequences with ground truth and real sequences illustrate our method. Comparisons with classical and recently published methods are also given to demonstrate the accuracy of our algorithm.     

Analog VLSI focal-plane array with dynamic connections for the estimation of piecewise-smooth optical flow

An analog very large-scale integrated (aVLSI) sensor is presented that is capable of estimating optical flow while detecting and preserving motion discontinuities. The sensor's architecture is composed of two recurrently connected networks. The units in the first network (the optical-flow network) collectively estimate two-dimensional optical flow, where the strength of their nearest-neighbor coupling determines the degree of motion integration. While the coupling strengths in our previous implementations were globally set and adjusted by the operator, they are now dynamically and locally controlled by a second on-chip network (the motion-discontinuity network). The coupling strengths are set such that visual motion integration is inhibited across image locations that are likely to represent motion boundaries. Results of a prototype sensor illustrate the potential of the approach and its functionality under real-world conditions.     

Rician envelope estimation and confidence intervals in low signal/noise levels

The determination of a Rician signal envelope in low signal/noise is a biased estimate of the true (noiseless) envelope. Four statistical estimators which attempt to correct for the biasing are investigated, namely the maximum likelihood, most probable, mean and median estimators. Confidence intervals on the envelope estimation are also presented at the 68%, 95% and 99% confidence levels.     

Measuring nonconstant flow in magnetic resonance imaging

An attempt is made to quantify rapid flow using magnetic resonance imaging techniques. An analysis is presented in which it is assumed that constant velocity gradients are present. A deconvolution scheme which can remove the blurring from motion with acceleration is developed. This allows improved resolution and velocity determination. Computer experiments were performed on simulated data, where the velocity drops from 100 cm/s to 50 cm/s over a distance of 5 cm. In noise-free data, velocities were recovered to within 2% of the lower velocity and, for data with 5% white noise, to within 6%     

Sparsity in optical flow and trajectories

In this article, we apply sparse constraints to improve optical flow and trajectories. We apply sparsity in two ways. First, with two-frame optical flow, we enforce a sparse representation of flow patches using a learned overcomplete dictionary. Second, we apply a low-rank constraint to trajectories via robust coupling. Optical flow is an ill-posed underconstrained inverse problem. Many recent approaches use total variation to constrain the flow solution to satisfy color constancy. In our first results presented, we find that learning a 2D overcomplete dictionary from the total variation result and then enforcing a sparse constraint on the flow improves the result. A new technique using partially overlapping patches accelerates the calculation. This approach is implemented in a coarse-to-fine strategy. Our results show that combining total variation and a sparse constraint from a learned dictionary is more effective than total variation alone. In the second part, we compute optical flow and trajectories from an image sequence. Sparsity in trajectories is measured by matrix rank. We introduce a low-rank constraint of linear complexity using random subsampling of the data. We demonstrate that, by using a robust coupling with the low-rank constraint, our approach outperforms baseline methods on general image sequences.     

Correlation-feedback technique in optical flow determination

We present a new algorithm to determine optical flow that utilizes a correlation-feedback technique. Several experiments are presented to demonstrate that our method performs generally better than some standard correlation and gradient-based methods in terms of accuracy.     

Asymptotic global confidence regions for 3-D parametric shape estimation in inverse problems.

This paper derives fundamental performance bounds for statistical estimation of parametric surfaces embedded in R3. Unlike conventional pixel-based image reconstruction approaches, our problem is reconstruction of the shape of binary or homogeneous objects. The fundamental uncertainty of such estimation problems can be represented by global confidenceregions, which facilitate geometric inference and optimization ofthe imaging system. Compared to our previous work on global confidence region analysis for curves [two-dimensional (2-D) shapes], computation of the probability that the entire surface estimate lies within the confidence region is more challenging because a surface estimate is an inhomogeneous random field continuously indexed by a 2-D variable. We derive an asymptotic lower bound to this probability by relating it to the exceedence probability of a higher dimensional Gaussian random field, which can, in turn, be evaluated using the tube formula due to Sun. Simulation results demonstrate the tightness of the resulting bound and the usefulness of the three-dimensional global confidence region approach.     

Fixed-width confidence interval estimation of the inverse coefficient of variation in a normal population

A sequential procedure is developed in order to construct a confidence interval of “fixed-width and preassigned coverage probability” for the inverse of the coefficient of variation of a normal population. The proposed sequential procedure is proved to be “asymptotically efficient and consistent” in the sense of Chow and Robbins ([1]: Ann. Math. Statist. 36, 457–462 (1965)). Asymptotic distribution of the stopping time is derived.