Time-Causal and Time-Recursive Spatio-Temporal Receptive Fields

We present an improved model and theory for time-causal and time-recursive spatio-temporal receptive fields, obtained by a combination of Gaussian receptive fields over the spatial domain and first-order integrators or equivalently truncated exponential filters coupled in cascade over the temporal domain. Compared to previous spatio-temporal scale-space formulations in terms of non-enhancement of local extrema or scale invariance, these receptive fields are based on different scale-space axiomatics over time by ensuring non-creation of new local extrema or zero-crossings with increasing temporal scale. Specifically, extensions are presented about (i) parameterizing the intermediate temporal scale levels, (ii) analysing the resulting temporal dynamics, (iii) transferring the theory to a discrete implementation in terms of recursive filters over time, (iv) computing scale-normalized spatio-temporal derivative expressions for spatio-temporal feature detection and (v) computational modelling of receptive fields in the lateral geniculate nucleus (LGN) and the primary visual cortex (V1) in biological vision. We show that by distributing the intermediate temporal scale levels according to a logarithmic distribution, we obtain a new family of temporal scale-space kernels with better temporal characteristics compared to a more traditional approach of using a uniform distribution of the intermediate temporal scale levels. Specifically, the new family of time-causal kernels has much faster temporal response properties (shorter temporal delays) compared to the kernels obtained from a uniform distribution. When increasing the number of temporal scale levels, the temporal scale-space kernels in the new family do also converge very rapidly to a limit kernel possessing true self-similar scale-invariant properties over temporal scales. Thereby, the new representation allows for true scale invariance over variations in the temporal scale, although the underlying temporal scale-space representation is based on a discretized temporal scale parameter. We show how scale-normalized temporal derivatives can be defined for these time-causal scale-space kernels and how the composed theory can be used for computing basic types of scale-normalized spatio-temporal derivative expressions in a computationally efficient manner.     

Spatio-Temporal Scale Selection in Video Data

This work presents a theory and methodology for simultaneous detection of local spatial and temporal scales in video data. The underlying idea is that if we process video data by spatio-temporal receptive fields at multiple spatial and temporal scales, we would like to generate hypotheses about the spatial extent and the temporal duration of the underlying spatio-temporal image structures that gave rise to the feature responses. For two types of spatio-temporal scale-space representations, (i) a non-causal Gaussian spatio-temporal scale space for offline analysis of pre-recorded video sequences and (ii) a time-causal and time-recursive spatio-temporal scale space for online analysis of real-time video streams, we express sufficient conditions for spatio-temporal feature detectors in terms of spatio-temporal receptive fields to deliver scale-covariant and scale-invariant feature responses. We present an in-depth theoretical analysis of the scale selection properties of eight types of spatio-temporal interest point detectors in terms of either: (i)–(ii) the spatial Laplacian applied to the first- and second-order temporal derivatives, (iii)–(iv) the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the spatio-temporal Laplacian and (vii)–(viii) the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix. It is shown that seven of these spatio-temporal feature detectors allow for provable scale covariance and scale invariance. Then, we describe a time-causal and time-recursive algorithm for detecting sparse spatio-temporal interest points from video streams and show that it leads to intuitively reasonable results. An experimental quantification of the accuracy of the spatio-temporal scale estimates and the amount of temporal delay obtained from these spatio-temporal interest point detectors is given, showing that: (i) the spatial and temporal scale selection properties predicted by the continuous theory are well preserved in the discrete implementation and (ii) the spatial Laplacian or the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives leads to much shorter temporal delays in a time-causal implementation compared to the determinant of the spatio-temporal Hessian or the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix.     

Feature Detection with Automatic Scale Selection

The fact that objects in the world appear in different ways depending on the scale of observation has important implications if one aims at describing them. It shows that the notion of scale is of utmost importance when processing unknown measurement data by automatic methods. In their seminal works, Witkin (1983) and Koenderink (1984) proposed to approach this problem by representing image structures at different scales in a so-called scale-space representation. Traditional scale-space theory building on this work, however, does not address the problem of how to select local appropriate scales for further analysis. This article proposes a systematic methodology for dealing with this problem. A framework is presented for generating hypotheses about interesting scale levels in image data, based on a general principle stating that local extrema over scales of different combinations of -normalized derivatives are likely candidates to correspond to interesting structures. Specifically, it is shown how this idea can be used as a major mechanism in algorithms for automatic scale selection, which adapt the local scales of processing to the local image structure.Support for the proposed approach is given in terms of a general theoretical investigation of the behaviour of the scale selection method under rescalings of the input pattern and by integration with different types of early visual modules, including experiments on real-world and synthetic data. Support is also given by a detailed analysis of how different types of feature detectors perform when integrated with a scale selection mechanism and then applied to characteristic model patterns. Specifically, it is described in detail how the proposed methodology applies to the problems of blob detection, junction detection, edge detection, ridge detection and local frequency estimation.In many computer vision applications, the poor performance of the low-level vision modules constitutes a major bottleneck. It is argued that the inclusion of mechanisms for automatic scale selection is essential if we are to construct vision systems to automatically analyse complex unknown environments.     

一种基于Gabor小波的局部特征尺度提取方法

图像的局部特征尺度在进行特征提取和构造尺度不变量时非常重要。提出了一种基于Gabor小波的局部特征尺度提取方法,该方法利用视皮层简单细胞的2维Gabor函数模型, 构造了一个Gabor尺度空间核函数,利用该核函数计算图像的Gabor尺度空间分解,并在尺度空间中搜索局部极大值作为特征点的固有尺度。实验结果表明,该方法可在不同对比度 条件下有效地提取各类特征的局部尺度,并且相比高斯拉普拉斯(LoG)方法有更好的适应性和可靠性。     

Scale Selection Properties of Generalized Scale-Space Interest Point Detectors

Scale-invariant interest points have found several highly successful applications in computer vision, in particular for image-based matching and recognition. This paper presents a theoretical analysis of the scale selection properties of a generalized framework for detecting interest points from scale-space features presented in Lindeberg (Int. J. Comput. Vis. 2010, under revision) and comprising:

an enriched set of differential interest operators at a fixed scale including the Laplacian operator, the determinant of the Hessian, the new Hessian feature strength measures I and II and the rescaled level curve curvature operator, as well as

an enriched set of scale selection mechanisms including scale selection based on local extrema over scale, complementary post-smoothing after the computation of non-linear differential invariants and scale selection based on weighted averaging of scale values along feature trajectories over scale.

It is shown how the selected scales of different linear and non-linear interest point detectors can be analyzed for Gaussian blob models. Specifically it is shown that for a rotationally symmetric Gaussian blob model, the scale estimates obtained by weighted scale selection will be similar to the scale estimates obtained from local extrema over scale of scale normalized derivatives for each one of the pure second-order operators. In this respect, no scale compensation is needed between the two types of scale selection approaches. When using post-smoothing, the scale estimates may, however, be different between different types of interest point operators, and it is shown how relative calibration factors can be derived to enable comparable scale estimates for each purely second-order operator and for different amounts of self-similar post-smoothing. A theoretical analysis of the sensitivity to affine image deformations is presented, and it is shown that the scale estimates obtained from the determinant of the Hessian operator are affine covariant for an anisotropic Gaussian blob model. Among the other purely second-order operators, the Hessian feature strength measure I has the lowest sensitivity to non-uniform scaling transformations, followed by the Laplacian operator and the Hessian feature strength measure II. The predictions from this theoretical analysis agree with experimental results of the repeatability properties of the different interest point detectors under affine and perspective transformations of real image data. A number of less complete results are derived for the level curve curvature operator.     

Scale-space for discrete signals

A basic and extensive treatment of discrete aspects of the scale-space theory is presented. A genuinely discrete scale-space theory is developed and its connection to the continuous scale-space theory is explained. Special attention is given to discretization effects, which occur when results from the continuous scale-space theory are to be implemented computationally. The 1D problem is solved completely in an axiomatic manner. For the 2D problem, the author discusses how the 2D discrete scale space should be constructed. The main results are as follows: the proper way to apply the scale-space theory to discrete signals and discrete images is by discretization of the diffusion equation, not the convolution integral; the discrete scale space obtained in this way can be described by convolution with the kernel, which is the discrete analog of the Gaussian kernel, a scale-space implementation based on the sampled Gaussian kernel might lead to undesirable effects and computational problems, especially at fine levels of scale; the 1D discrete smoothing transformations can be characterized exactly and a complete catalogue is given; all finite support 1D discrete smoothing transformations arise from repeated averaging over two adjacent elements (the limit case of such an averaging process is described); and the symmetric 1D discrete smoothing kernels are nonnegative and unimodal, in both the spatial and the frequency domain     

A syntactic approach to scale-space-based corner description

Planar curves are described by information about corners integrated over various levels of resolution. The detection of corners takes place on a digital representation. To compensate for ambiguities arising from sampling problems due to the discreteness, results about the local behavior of curvature extrema in continuous scale-space are employed     

Non-Linear Scale-Spaces Isomorphic to the Linear Case with Applications to Scalar,Vector and Multispectral Images

A basic requirement of scale-space representations in general is that of scale causality, which states that local extrema in the image should not be enhanced when resolution is diminished. We consider a special class of nonlinear scale-spaces consistent with this constraint, which can be linearised by a suitable isomorphism in the grey-scale domain so as to reproduce the familiar Gaussian scale-space. We consider instances in which nonlinear representations may be the preferred choice, as well as instances in which they enter by necessity. We also establish their relation to morphological scale-space representations based on a quadratic structuring function.     

尺度与颜色不变性图像特征描述

尺度不变特征变换是目前公认的鲁棒性最强的图像特征描述方法之一,在尺度不变性和几何不变性方面具有较好的特性,但该方法主要适用于灰度图像,对图像颜色的区分能力不强,因此,一些对象可能会因为颜色的不同而被错误的区分.另外,尺度不变特征变换对关键点局部范围内描述子主方向的依赖性非常强,直接决定了匹配的正确率,但是研究表明,主方向分配产生的误差仅有三分之二左右能控制在[-20。,+20。]范围内,因此部分特征会有三分之一的概率因为主方向分配的误差较大而不能正确匹配.针对以上两个问题,本文提出了一种具有颜色和尺度不变性的局部特征描述方法,颜色不变性通过将RGB图像转换到高斯颜色模型下实现,特征描述过程中不再分配主方向,而用局部相对方向,尺度不变性通过构建高斯金子塔实现.实验选取阿姆斯特丹数据集图像进行了测试,结果表明本文方法比传统尺度不变特征变换方法,在特征点的数目、分布均匀性以及匹配精度方面均有所提高.     

Scale-space properties of the multiscale morphologicaldilation-erosion

A multiscale morphological dilation-erosion smoothing operation and its associated scale-space expansion for multidimensional signals are proposed. Properties of this smoothing operation are developed and, in particular a scale-space monotonic property for signal extrema is demonstrated. Scale-space fingerprints from this approach have advantages over Gaussian scale-space fingerprints in that: they are defined for negative values of the scale parameter; have monotonic properties in two and higher dimensions; do not cause features to be shifted by the smoothing; and allow efficient computation. The application of reduced multiscale dilation-erosion fingerprints to the surface matching of terrain is demonstrated     

基于视觉特征信息量度量的高斯尺度参数自适应算法

为了避免计算过于复杂或因丢弃过多关键信息而造成失真过大的问题,在高斯尺度空间的构造中应正确选用尺度参数,以使图像信息的变化呈现均匀的特点。目前,许多高斯尺度空间应用中采用的层之间的尺度参数关系并不明确,使得分层效果不理想。本文基于视觉特征模型,提出一种自适应高斯尺度参数的算法,并通过对SAR图像降噪处理对比试验验证了它的有效性,从而为图像的高层次处理如目标识别等提供了信息量稳定变化的尺度空间。     

Effective scale: a natural unit for measuring scale-space lifetime

A manner in which a notion of effective scale can be introduced in a formal way is developed. For continuous signals, a scaling argument directly gives a natural unit for measuring scale-space lifetime in terms of the logarithm of the ordinary scale parameter. That approach is, however, not appropriate for discrete signals since an infinite lifetime would be assigned to structures existing in the original signal. It is shown how such an effective scale parameter can be defined to give consistent results for both discrete and continuous signals. The treatment is based on the assumption that the probability that a local extremum disappears during a short-scale interval should not vary with scale. As a tool for the analysis, estimates are given of how the density of local extrema can be expected to vary with scale in the scale-space representation of different random noise signals both in the continuous and discrete cases     

Adaptive determination of filter scales for edge detection

The authors suggest a regularization method for determining scales for edge detection adaptively for each site in the image plane. Specifically, they extend the optimal filter concept of T. Poggio et al. (1984) and the scale-space concept of A. Witkin (1983) to an adaptive scale parameter. To avoid an ill-posed feature synthesis problem, the scheme automatically finds optimal scales adaptively for each pixel before detecting final edge maps. The authors introduce an energy function defined as a functional over continuous scale space. Natural constraints for edge detection are incorporated into the energy function. To obtain a set of optimal scales that can minimize the energy function, a parallel relaxation algorithm is introduced. Experiments for synthetic and natural scenes show the advantages of the algorithm. In particular, it is shown that this system can detect both step and diffuse edges while drastically filtering out the random noise     

An Analysis of the Factors Affecting Keypoint Stability in Scale-Space

The most popular image matching algorithm SIFT, introduced by D. Lowe a decade ago, has proven to be sufficiently scale invariant to be used in numerous applications. In practice, however, scale invariance may be weakened by various sources of error inherent to the SIFT implementation affecting the stability and accuracy of keypoint detection. The density of the sampling of the Gaussian scale-space and the level of blur in the input image are two of these sources. This article presents a numerical analysis of their impact on the extracted keypoints stability. Such an analysis has both methodological and practical implications, on how to compare feature detectors and on how to improve SIFT. We show that even with a significantly oversampled scale-space numerical errors prevent from achieving perfect stability. Usual strategies to filter out unstable detections (e.g., poorly contrasted extrema) are shown to be inefficient. We also prove that the effect of the error in the assumption on the initial blur is asymmetric and that the method is strongly degraded in the presence of aliasing or without a correct assumption on the camera blur. This analysis leads to a series of practical recommendations.     

Scale-space from nonlinear filters

Decomposition by extrema is put into the context of linear vision systems and scale-space. It is proved that discrete one-dimensional, M- and N-sieves neither introduce new edges as the scale increases nor create new extrema. They share this property with diffusion based filters. They are robust and preserve edges of large scale features