一种改进的变分光流计算方法

提出一种改进的变分光流算法的能量泛函,该能量泛函的数据项由灰度不变假设和Hessian矩阵不变假设组成,并与Lucas局部光流一致方法相结合。平滑项的设计采用先各项同性平滑再各项异性平滑的策略,其中引入图像一致增强思想。实验结果证明,运用该方法进行光流计算的效果比以往变分方法有所改进。     

铁路视频序列的FOE的估计

在结构化场景的轨道交通中,车载视频观测因相机平移运动而呈现出图像内容以某点为中心向四周扩散的现象,该点被称为FOE (Focus of Expansion)。 当前计算FOE的算法对噪声敏感且计算量大,不能准确地计算铁路场景中的FOE。鉴于此,文中提出一种铁路视频序列的FOE估计方法。该方法首先利用金字塔光流法对检测的Harris角点进行跟踪和粗匹配,并在此基础上利用RANSAC算法进行精确的匹配,求得基础矩阵,然后提取图像中的极线束并计算FOE。实验结果表明,所提算法比Hough 直线求得的FOE误差小,适于实时应用。     

Tri-view morphing

This paper presents an efficient image-based approach to navigate a scene based on only three wide-baseline uncalibrated images without the explicit use of a 3D model. After automatically recovering corresponding points between each pair of images, an accurate trifocal plane is extracted from the trifocal tensor of these three images. Next, based on a small number of feature marks using a friendly GUI, the correct dense disparity maps are obtained by using our trinocular-stereo algorithm. Employing the barycentric warping scheme with the computed disparity, we can generate an arbitrary novel view within a triangle spanned by three camera centers. Furthermore, after self-calibration of the cameras, 3D objects can be correctly augmented into the virtual environment synthesized by the tri-view morphing algorithm. Three applications of the tri-view morphing algorithm are demonstrated. The first one is 4D video synthesis, which can be used to fill in the gap between a few sparsely located video cameras to synthetically generate a video from a virtual moving camera. This synthetic camera can be used to view the dynamic scene from a novel view instead of the original static camera views. The second application is multiple view morphing, where we can seamlessly fly through the scene over a 2D space constructed by more than three cameras. The last one is dynamic scene synthesis using three still images, where several rigid objects may move in any orientation or direction. After segmenting three reference frames into several layers, the novel views in the dynamic scene can be generated by applying our algorithm. Finally, the experiments are presented to illustrate that a series of photo-realistic virtual views can be generated to fly through a virtual environment covered by several static cameras.     

Shot Change Detection Using Scene-Based Constraint

A key step for managing a large video database is to partition the video sequences into shots. Past approaches to this problem tend to confuse gradual shot changes with changes caused by smooth camera motions. This is in part due to the fact that camera motion has not been dealt with in a more fundamental way. We propose an approach that is based on a physical constraint used in optical flow analysis, namely, the total brightness of a scene point across two frames should remain constant if the change across two frames is a result of smooth camera motion. Since the brightness constraint would be violated across a shot change, the detection can be based on detecting the violation of this constraint. It is robust because it uses only the qualitative aspect of the brightness constraint—detecting a scene change rather than estimating the scene itself. Moreover, by tapping on the significant know-how in using this constraint, the algorithm's robustness is further enhanced. Experimental results are presented to demonstrate the performance of various algorithms. It was shown that our algorithm is less likely to interpret gradual camera motions as shot changes, resulting in a significantly better precision performance than most other algorithms.     

Highly Accurate Optic Flow Computation with Theoretically Justified Warping

In this paper, we suggest a variational model for optic flow computation based on non-linearised and higher order constancy assumptions. Besides the common grey value constancy assumption, also gradient constancy, as well as the constancy of the Hessian and the Laplacian are proposed. Since the model strictly refrains from a linearisation of these assumptions, it is also capable to deal with large displacements. For the minimisation of the rather complex energy functional, we present an efficient numerical scheme employing two nested fixed point iterations. Following a coarse-to-fine strategy it turns out that there is a theoretical foundation of so-called warping techniques hitherto justified only on an experimental basis. Since our algorithm consists of the integration of various concepts, ranging from different constancy assumptions to numerical implementation issues, a detailed account of the effect of each of these concepts is included in the experimental section. The superior performance of the proposed method shows up by significantly smaller estimation errors when compared to previous techniques. Further experiments also confirm excellent robustness under noise and insensitivity to parameter variations.     

Dynamic Texture Detection Based on Motion Analysis

Motion estimation is usually based on the brightness constancy assumption. This assumption holds well for rigid objects with a Lambertian surface, but it is less appropriate for fluid and gaseous materials. For these materials an alternative assumption is required. This work examines three possible alternatives: gradient constancy, color constancy and brightness conservation (under this assumption the brightness of an object can diffuse to its neighborhood). Brightness conservation and color constancy are found to be adequate models. We propose a method for detecting regions of dynamic texture in image sequences. Accurate segmentation into regions of static and dynamic texture is achieved using a level set scheme. The level set function separates each image into regions that obey brightness constancy and regions that obey the alternative assumption. We show that the method can be simplified to obtain a less robust but fast algorithm, capable of real-time performance. Experimental results demonstrate accurate segmentation by the full level set scheme, as well as by the simplified method. The experiments included challenging image sequences, in which color or geometry cues by themselves would be insufficient.     

A reality check for radiometric camera response recovery algorithms

The radiometric response of a camera governs the relationship between the incident light on the camera sensor and the output pixel values that are produced. This relationship, which is typically unknown and nonlinear, needs to be estimated for applications that require accurate measurement of scene radiance. Until now, various camera response recovery algorithms have been proposed each with different merits and drawbacks. However, an evaluation study that compares these algorithms has not been presented. In this work, we aim to fill this gap by conducting a rigorous experiment that evaluates the selected algorithms with respect to three metrics: consistency, accuracy, and robustness. In particular, we seek the answer of the following four questions: (1) Which camera response recovery algorithm gives the most accurate results? (2) Which algorithm produces the camera response most consistently for different scenes? (3) Which algorithm performs better under varying degrees of noise? (4) Does the sRGB assumption hold in practice? Our findings indicate that Grossberg and Nayar's (GN) algorithm (2004 [1]) is the most accurate; Mitsunaga and Nayar's (MN) algorithm (1999 [2]) is the most consistent; and Debevec and Malik's (DM) algorithm (1997 [3]) is the most resistant to noise together with MN. We also find that the studied algorithms are not statistically better than each other in terms of accuracy although all of them statistically outperform the sRGB assumption. By answering these questions, we aim to help the researchers and practitioners in the high dynamic range (HDR) imaging community to make better choices when choosing an algorithm for camera response recovery.     

Method for HDR-like imaging using industrial digital cameras

Dynamic range of the scene can be significantly wider than the dynamic range of an image because of limitations of A/D conversion. In such a situation, numerous details of the scene cannot be adequately shown on the image. Standard industrial digital cameras are equipped with an auto-exposure function that automatically sets both the aperture value and cameras exposure time. When measuring a scene with atypical distribution of light and dark elements, the indicated auto-exposure time may not be optimal. The aim of work was to improve, with minimal cost, the performance of standard industrial digital cameras. We propose a low complexity method for creating HDR-like image using three images captured with different exposure times. The proposed method consists of three algorithms: (1) algorithm for estimating whether the auto-exposure time is optimal, (2) algorithm which determines exposure times for two additional images (one with shorter and another with longer than auto-exposure time), and (3) algorithm for HDR-like imaging based on fusion of three previously obtained images. Method is implemented on FPGA inserted into standard industrial digital camera. Results show that the proposed approach produces high quality HDR-like scene-mapped 8-bit images with minimal computational cost. All improvements may be noticed through the performance evaluation.

     

Multi-view Scene Flow Estimation: A View Centered Variational Approach

We present a novel method for recovering the 3D structure and scene flow from calibrated multi-view sequences. We propose a 3D point cloud parametrization of the 3D structure and scene flow that allows us to directly estimate the desired unknowns. A unified global energy functional is proposed to incorporate the information from the available sequences and simultaneously recover both depth and scene flow. The functional enforces multi-view geometric consistency and imposes brightness constancy and piecewise smoothness assumptions directly on the 3D unknowns. It inherently handles the challenges of discontinuities, occlusions, and large displacements. The main contribution of this work is the fusion of a 3D representation and an advanced variational framework that directly uses the available multi-view information. This formulation allows us to advantageously bind the 3D unknowns in time and space. Different from optical flow and disparity, the proposed method results in a nonlinear mapping between the images’ coordinates, thus giving rise to additional challenges in the optimization process. Our experiments on real and synthetic data demonstrate that the proposed method successfully recovers the 3D structure and scene flow despite the complicated nonconvex optimization problem.     

Color by correlation: a simple, unifying framework for colorconstancy

The paper considers the problem of illuminant estimation: how, given an image of a scene, recorded under an unknown light, we can recover an estimate of that light. Obtaining such an estimate is a central part of solving the color constancy problem. Thus, the work presented will have applications in fields such as color-based object recognition and digital photography. Rather than attempting to recover a single estimate of the illuminant, we instead set out to recover a measure of the likelihood that each of a set of possible illuminants was the scene illuminant. We begin by determining which image colors can occur (and how these colors are distributed) under each of a set of possible lights. We discuss how, for a given camera, we can obtain this knowledge. We then correlate this information with the colors in a particular image to obtain a measure of the likelihood that each of the possible lights was the scene illuminant. Finally, we use this likelihood information to choose a single light as an estimate of the scene illuminant. Computation is expressed and performed in a generic correlation framework which we develop. We propose a new probabilistic instantiation of this correlation framework and show that it delivers very good color constancy on both synthetic and real images. We further show that the proposed framework is rich enough to allow many existing algorithms to be expressed within it: the gray-world and gamut-mapping algorithms are presented in this framework and we also explore the relationship of these algorithms to other probabilistic and neural network approaches to color constancy     

Robust computation of optical flow in a multi-scale differential framework

We have developed a new algorithm for computing optical flow in a differential framework. The image sequence is first convolved with a set of linear, separable spatiotemporal filter kernels similar to those that have been used in other early vision problems such as texture and stereopsis. The brightness constancy constraint can then be applied to each of the resulting images, giving us, in general, an overdetermined system of equations for the optical flow at each pixel. There are three principal sources of error: (a) stochastic error due to sensor noise (b) systematic errors in the presence of large displacements and (c) errors due to failure of the brightness constancy model. Our analysis of these errors leads us to develop an algorithm based on a robust version of total least squares. Each optical flow vector computed has an associated reliability measure which can be used in subsequent processing. The performance of the algorithm on the data set used by Barron et al. (IJCV 1994) compares favorably with other techniques. In addition to being separable, the filters used are also causal, incorporating only past time frames. The algorithm is fully parallel and has been implemented on a multiple processor machine.     

Independent motion detection in 3D scenes

This paper presents an algorithmic approach to the problem of detecting independently moving objects in 3D scenes that are viewed under camera motion. There are two fundamental constraints that can be exploited for the problem: 1) two/multiview camera motion constraint (for instance, the epipolar/trilinear constraint) and 2) shape constancy constraint. Previous approaches to the problem either use only partial constraints, or rely on dense correspondences or flow. We employ both the fundamental constraints in an algorithm that does not demand a priori availability of correspondences or flow. Our approach uses the plane-plus-parallax decomposition to enforce the two constraints. It is also demonstrated that for a class of scenes, called sparse 3D scenes in which genuine parallax and independent motions may be confounded, how the plane-plus-parallax decomposition allows progressive introduction, and verification of the fundamental constraints. Results of the algorithm on some difficult sparse 3D scenes are promising.     

Determining shape and motion from non-overlapping multi-camera rig: A direct approach using normal flows

In this paper, we explore how a wide field-of-view imaging system that consists of a number of cameras in a network arranged to approximate a spherical eye can reduce the complexity of estimating camera motion. Depth map of the imaged scene can be reconstructed once the camera motion is there. We present a direct method to recover camera motion from video data, which neither requires establishment of feature correspondences nor recovery of optical flow, but from normal flow which is directly observable. With a wide visual field, the inherent ambiguities between translation and rotation disappear. Several subsets of normal flow pairs and triplets can be utilized to constraint the directions of translation and rotation separately. The intersection of solution spaces arising from normal flow pairs or triplets yields the estimate on the direction of motion. In addition, the larger number of normal flow measurements so resulted can be used to combat the local flow extraction error. Rotational magnitude is recovered in a subsequent stage. This article details how motion recovery can be improved with the use of such an approximate spherical imaging system. Experimental results on synthetic and real image data are provided. The results show that the accuracy of motion estimation is comparable to those of the state-of-the-art methods that require to use explicit feature correspondences or full optical flows, and our method has a much faster computational speed.     

A Theoretical Framework for Convex Regularizers in PDE-Based Computation of Image Motion

Many differential methods for the recovery of the optic flow field from an image sequence can be expressed in terms of a variational problem where the optic flow minimizes some energy. Typically, these energy functionals consist of two terms: a data term, which requires e.g. that a brightness constancy assumption holds, and a regularizer that encourages global or piecewise smoothness of the flow field. In this paper we present a systematic classification of rotation invariant convex regularizers by exploring their connection to diffusion filters for multichannel images. This taxonomy provides a unifying framework for data-driven and flow-driven, isotropic and anisotropic, as well as spatial and spatio-temporal regularizers. While some of these techniques are classic methods from the literature, others are derived here for the first time. We prove that all these methods are well-posed: they posses a unique solution that depends in a continuous way on the initial data. An interesting structural relation between isotropic and anisotropic flow-driven regularizers is identified, and a design criterion is proposed for constructing anisotropic flow-driven regularizers in a simple and direct way from isotropic ones. Its use is illustrated by several examples.     

Distortion Optimization based Image Completion from a Large Displacement View

We present a new image completion method based on an additional large displacement view (LDV) of the same scene for faithfully repairing large missing regions on the target image in an automatic way. A coarse‐to‐fine distortion correction algorithm is proposed to minimize the perspective distortion in the corresponding parts for the common scene regions on the LDV image. First, under the assumption of a planar scene, the LDV image is warped according to a homography to generate the initial correction result. Second, the residual distortions in the common known scene regions are revealed by means of a mismatch detection mechanism and relaxed by energy optimization of overlap correspondences, with the expectations of color constancy and displacement field smoothness. The fundamental matrix for the two views is then computed based on the reliable correspondence set. Third, under the constraints of epipolar geometry, displacement field smoothness and color consistency of the neighboring pixels, the missing pixels are orderly restored according to a specially defined repairing priority function. We finally eliminate the ghost effect between the repaired region and its surroundings by Poisson image blending. Experimental results demonstrate that our method outperforms recent state‐of‐the‐art image completion methods for repairing large missing area with complex structure information.     

Linearized motion estimation for articulated planes

In this paper, we describe the explicit application of articulation constraints for estimating the motion of a system of articulated planes. We relate articulations to the relative homography between planes and show that these articulations translate into linearized equality constraints on a linear least-squares system, which can be solved efficiently using a Karush-Kuhn-Tucker system. The articulation constraints can be applied for both gradient-based and feature-based motion estimation algorithms and to illustrate this, we describe a gradient-based motion estimation algorithm for an affine camera and a feature-based motion estimation algorithm for a projective camera that explicitly enforces articulation constraints. We show that explicit application of articulation constraints leads to numerically stable estimates of motion. The simultaneous computation of motion estimates for all of the articulated planes in a scene allows us to handle scene areas where there is limited texture information and areas that leave the field of view. Our results demonstrate the wide applicability of the algorithm in a variety of challenging real-world cases such as human body tracking, motion estimation of rigid, piecewise planar scenes, and motion estimation of triangulated meshes.     

Rigid body segmentation and shape description from dense opticalflow under weak perspective

We present an algorithm for identifying and tracking independently moving rigid objects from optical flow. Some previous attempts at segmentation via optical flow have focused on finding discontinuities in the flow field. While discontinuities do indicate a change in scene depth, they do not in general signal a boundary between two separate objects. The proposed method uses the fact that each independently moving object has a unique epipolar constraint associated with its motion. Thus motion discontinuities based on self-occlusion can be distinguished from those due to separate objects. The use of epipolar geometry allows for the determination of individual motion parameters for each object as well as the recovery of relative depth for each point on the object. The algorithm assumes an affine camera where perspective effects are limited to changes in overall scale. No camera calibration parameters are required. A Kalman filter based approach is used for tracking motion parameters with time     

Real-time joint disparity and disparity flow estimation on programmable graphics hardware

Disparity flow depicts the 3D motion of a scene in the disparity space of a given view and can be considered as view-dependent scene flow. A novel algorithm is presented to compute disparity maps and disparity flow maps in an integrated process. Consequently, the disparity flow maps obtained helps to enforce the temporal consistency between disparity maps of adjacent frames. The disparity maps found also provides the spatial correspondence information that can be used to cross-validate disparity flow maps of different views. Two different optimization approaches are integrated in the presented algorithm for searching optimal disparity values and disparity flows. The local winner-take-all approach runs faster, whereas the global dynamic programming based approach produces better results. All major computations are performed in the image space of the given view, leading to an efficient implementation on programmable graphics hardware. Experimental results on captured stereo sequences demonstrate the algorithm’s capability of estimating both 3D depth and 3D motion in real-time. Quantitative performance evaluation using synthetic data with ground truth is also provided.     

Structure and motion from line segments in multiple images

This paper presents a new method for recovering the three dimensional structure of a scene composed of straight line segments using the image data obtained from a moving camera. The recovery algorithm is formulated in terms of an objective function which measures the total squared distance in the image plane between the observed edge segments and the projections (perspective) of the reconstructed lines. This objective function is minimized with respect to the line parameters and the camera positions to obtain an estimate for the structure of the scene. The effectiveness of this approach is demonstrated quantitatively through extensive simulations and qualitatively with actual image sequences. The implementation is being made publicly available     

Two-View Orthographic Epipolar Geometry: Minimal and Optimal Solvers

We will in this paper present methods and algorithms for estimating two-view geometry based on an orthographic camera model. We use a previously neglected nonlinear criterion on rigidity to estimate the calibrated essential matrix. We give efficient algorithms for estimating it minimally (using only three point correspondences), in a least squares sense (using four or more point correspondences), and optimally with respect to the number of inliers. The inlier-optimal algorithm is based on a three-point solver and gives a fourth-order polynomial time algorithm. These methods can be used as building blocks to robustly find inlier correspondences in the presence of high degrees of outliers. We show experimentally that our methods can be used in many instances, where the orthographic camera model isn’t generally used. A case of special interest is situations with repetitive structures, which give high amounts of outliers in the initial feature point matching.