Subspace information criterion for model selection

The problem of model selection is considerably important for acquiring higher levels of generalization capability in supervised learning. In this article, we propose a new criterion for model selection, the subspace information criterion (SIC), which is a generalization of Mallows's C(L). It is assumed that the learning target function belongs to a specified functional Hilbert space and the generalization error is defined as the Hilbert space squared norm of the difference between the learning result function and target function. SIC gives an unbiased estimate of the generalization error so defined. SIC assumes the availability of an unbiased estimate of the target function and the noise covariance matrix, which are generally unknown. A practical calculation method of SIC for least-mean-squares learning is provided under the assumption that the dimension of the Hilbert space is less than the number of training examples. Finally, computer simulations in two examples show that SIC works well even when the number of training examples is small.     

L_(1/2) regularization

In this paper we propose an L1 /2 regularizer which has a nonconvex penalty.The L 1/2 regularizer is shown to have many promising properties such as unbiasedness, sparsity and oracle properties.A reweighed iterative algorithm is proposed so that the solution of the L 1/2 regularizer can be solved through transforming it into the solution of a series of L 1 regularizers.The solution of the L 1/2 regularizer is more sparse than that of the L 1 regularizer, while solving the L 1/2 regularizer is much simpler t...     

A fault-tolerant regularizer for RBF networks.

In classical training methods for node open fault, we need to consider many potential faulty networks. When the multinode fault situation is considered, the space of potential faulty networks is very large. Hence, the objective function and the corresponding learning algorithm would be computationally complicated. This paper uses the Kullback-Leibler divergence to define an objective function for improving the fault tolerance of radial basis function (RBF) networks. With the assumption that there is a Gaussian distributed noise term in the output data, a regularizer in the objective function is identified. Finally, the corresponding learning algorithm is developed. In our approach, the objective function and the learning algorithm are computationally simple. Compared with some conventional approaches, including weight-decay-based regularizers, our approach has a better fault-tolerant ability. Besides, our empirical study shows that our approach can improve the generalization ability of a fault-free RBF network.     

A Theoretical Framework for Convex Regularizers in PDE-Based Computation of Image Motion

Many differential methods for the recovery of the optic flow field from an image sequence can be expressed in terms of a variational problem where the optic flow minimizes some energy. Typically, these energy functionals consist of two terms: a data term, which requires e.g. that a brightness constancy assumption holds, and a regularizer that encourages global or piecewise smoothness of the flow field. In this paper we present a systematic classification of rotation invariant convex regularizers by exploring their connection to diffusion filters for multichannel images. This taxonomy provides a unifying framework for data-driven and flow-driven, isotropic and anisotropic, as well as spatial and spatio-temporal regularizers. While some of these techniques are classic methods from the literature, others are derived here for the first time. We prove that all these methods are well-posed: they posses a unique solution that depends in a continuous way on the initial data. An interesting structural relation between isotropic and anisotropic flow-driven regularizers is identified, and a design criterion is proposed for constructing anisotropic flow-driven regularizers in a simple and direct way from isotropic ones. Its use is illustrated by several examples.     

基于Mask R-CNN改进的遥感图像舰船检测

遥感图像中舰船朝向不确定性,舰船种类的多样性以及和其他海上及港口物体之间的相似性,使舰船检测的性能下降严重。针对这一问题,使用一种简单且有效的方法来训练有旋转不变性和Fisher判别的Mask R-CNN舰船检测模型,通过优化模型的目标函数以提高舰船检测性能,在保持原有检测模型结构不变的基础上引入两个正则化器,第一个正则化器加强训练样本旋转之前和之后的特征联系,第二个正则化器限制卷积神经网络有小的类内散度和大的类间散度。实验中,在Kaggle遥感图像船只检测数据集上验证了所提出的方法提高了检测遥感图像中舰船目标的性能。     

一种自适应混合权重的自步学习方法

自步学习是一种受人类和动物学习过程启发的学习机制,它赋予训练样本不同的权重,从而逐步将简单到更复杂的样本纳入训练集进行学习.自步学习在目标函数中加入自步正则项控制学习过程.目前存在多种形式的自步权重正则项,不同的正则项可能会导致不同的学习性能.其中,混合权重正则项同时具有硬权重和软权重的特点,因而被广泛应用在众多自步学习问题中.然而,当前的混合权重方法只结合了对数软权重,形式较为单一.此外,相较于软权重或硬权重方式,混合权重方法引入了更多的参数.提出一种自适应混合权重的自步正则方法来克服形式单一和参数难以调节的问题.一方面,在学习的过程中权重的表示形式能够自适应进行调整,另一方面,可以根据样本损失分布特点来自适应混合权重引入的自步参数,从而减少参数对人为经验的依赖.行为识别和多媒体事件检测上的实验结果表明提出的方法可以有效地解决权重形式和参数的自适应问题.     

A dynamical system perspective of structural learning withforgetting

Structural learning with forgetting is an established method of using Laplace regularization to generate skeletal artificial neural networks. We develop a continuous dynamical system model of regularization in which the associated regularization parameter is generalized to be a time-varying function. Analytic results are obtained for a Laplace regularizer and a quadratic error surface by solving a different linear system in each region of the weight space. This model also enables a comparison of Laplace and Gaussian regularization. Both of these regularizers have a greater effect in weight space directions which are less important for minimization of a quadratic error function. However, for the Gaussian regularizer, the regularization parameter modifies the associated linear system eigenvalues, in contrast to its function as a control input in the Laplace case. This difference provides additional evidence for the superiority of the Laplace over the Gaussian regularizer.     

几种矢量图像噪声去除变分模型的边缘保持比较

矢量图像噪声去除的变分模型必须考虑不同通道图像间的耦合以保持图像边缘,但所得到的模型复杂、计算效率低,且不同耦合方法对应的模型的边缘保持质量不同。本文首先设计了目前已经提出的这类变分模型的快速Split Bregman算法,然后通过大量数值实验对不同模型的边缘保持特性和计算效率进行了比较。所研究的模型分别使用LTV(layered total variation)规则项、MTV(multichannel total variation)规则项、CTV(color total variation)规则项、PA(polyakov action)规则项和RPA(reduced polyakov action)规则项。实验结果表明CTV模型对矢量图像去噪边缘保持最好,其他依次是PA模型、MTV模型、RPA模型和LTV模型;LTV模型计算效率最高,其他依次是MTV模型、RPA模型、CTV模型和PA模型。     

Network information criterion-determining the number of hiddenunits for an artificial neural network model

The problem of model selection, or determination of the number of hidden units, can be approached statistically, by generalizing Akaike's information criterion (AIC) to be applicable to unfaithful (i.e., unrealizable) models with general loss criteria including regularization terms. The relation between the training error and the generalization error is studied in terms of the number of the training examples and the complexity of a network which reduces to the number of parameters in the ordinary statistical theory of AIC. This relation leads to a new network information criterion which is useful for selecting the optimal network model based on a given training set.     

稀疏正则化逆向神经网络在双陷波超宽带天线设计中的应用

在双陷波超宽带天线的设计过程中,直接逆向神经网络模型精度较低,而 BP 逆向神经网络泛化能力较差,若单独使用 HFSS 仿真软件需要不断优化天线各参数增加了设计时间。 针对以上问题,提出一种将 HFSS 与稀疏正则化逆向神经网络联合的方法。 该方法在逆向神经网络性能函数中增加 l1/2范数和 l2 范数,l1/2 范数引入了新的权系数,扩充了输入样本向量,使网络更易得到稀疏性解,逆模型精度更高,l2 范数能有效避免过拟合现象,使网络泛化能力更强。 应用于双陷波超宽带天线设计中,采用在辐射贴片上开弧形槽的方式产生陷波特性,根据天线目标电压驻波比逆向求解对应的开槽尺寸。 仿真实验结果表明,与 BP 逆向神经网络方法相比,求得的与天线电压驻波比对应的开槽角度相对误差减小了 69.3% ,开槽半径相对误差减小了 88.7% ,网络运行时间减少了 15.9% ;最终设计的天线带宽为 2.4~11GHz ,实现了3.31~3.8GHz 和4.98~6.05GHz 的良好陷波特性,缩短了整个天线的设计周期。     

Trading variance reduction with unbiasedness: the regularized subspace information criterion for robust model selection in kernel regression

A well-known result by Stein (1956) shows that in particular situations, biased estimators can yield better parameter estimates than their generally preferred unbiased counterparts. This letter follows the same spirit, as we will stabilize the unbiased generalization error estimates by regularization and finally obtain more robust model selection criteria for learning. We trade a small bias against a larger variance reduction, which has the beneficial effect of being more precise on a single training set. We focus on the subspace information criterion (SIC), which is an unbiased estimator of the expected generalization error measured by the reproducing kernel Hilbert space norm. SIC can be applied to the kernel regression, and it was shown in earlier experiments that a small regularization of SIC has a stabilization effect. However, it remained open how to appropriately determine the degree of regularization in SIC. In this article, we derive an unbiased estimator of the expected squared error, between SIC and the expected generalization error and propose determining the degree of regularization of SIC such that the estimator of the expected squared error is minimized. Computer simulations with artificial and real data sets illustrate that the proposed method works effectively for improving the precision of SIC, especially in the high-noise-level cases. We furthermore compare the proposed method to the original SIC, the cross-validation, and an empirical Bayesian method in ridge parameter selection, with good results.     

A novel l 1/2 sparse regression method for hyperspectral unmixing

Hyperspectral unmixing (HU) is a popular tool in remotely sensed hyperspectral data interpretation, and it is used to estimate the number of reference spectra (end-members), their spectral signatures, and their fractional abundances. However, it can also be assumed that the observed image signatures can be expressed in the form of linear combinations of a large number of pure spectral signatures known in advance (e.g. spectra collected on the ground by a field spectro-radiometer, called a spectral library). Under this assumption, the solution of the fractional abundances of each spectrum can be seen as sparse, and the HU problem can be modelled as a constrained sparse regression (CSR) problem used to compute the fractional abundances in a sparse (i.e. with a small number of terms) linear mixture of spectra, selected from large libraries. In this article, we use the l _1/2 regularizer with the properties of unbiasedness and sparsity to enforce the sparsity of the fractional abundances instead of the l ₀ and l ₁ regularizers in CSR unmixing models, as the l _1/2 regularizer is much easier to be solved than the l ₀ regularizer and has stronger sparsity than the l ₁ regularizer (Xu et al. 2010). A reweighted iterative algorithm is introduced to convert the l _1/2 problem into the l ₁ problem; we then use the Split Bregman iterative algorithm to solve this reweighted l ₁ problem by a linear transformation. The experiments on simulated and real data both show that the l _1/2 regularized sparse regression method is effective and accurate on linear hyperspectral unmixing.     

并行效率敏感的大规模SVM数据分块数选择

数据分块数的选择是并行/分布式机器学习模型选择的基本问题之一,直接影响着机器学习算法的泛化性和运行效率。现有并行/分布式机器学习方法往往根据经验或处理器个数来选择数据分块数,没有明确的数据分块数选择准则。提出一个并行效率敏感的并行/分布式机器学习数据分块数选择准则,该准则可在保证并行/分布式机器学习模型测试精度的情况下,提高计算效率。首先推导并行/分布式机器学习模型的泛化误差与分块数目的关系。然后以此为基础,提出折衷泛化性与并行效率的数据分块数选择准则。最后,在ADMM框架下随机傅里叶特征空间中,给出采用该数据分块数选择准则的大规模支持向量机实现方案,并在高性能计算集群和大规模标准数据集上对所提出的数据分块数选择准则的有效性进行实验验证。     

Convex multiclass segmentation with shearlet regularization

Segmentation plays an important role in many preprocessing stages in image processing. Recently, convex relaxation methods for image multi-labelling were proposed in the literature. Often these models involve the total variation (TV) semi-norm as regularizing term. However, it is well known that the TV-functional is not optimal for the segmentation of textured regions. In recent years, directional representation systems were proposed to cope with curved singularities in images. In particular, curvelets and shearlets provide an optimally sparse approximation in the class of piecewise smooth functions with C ² singularity boundaries. In this paper, we demonstrate that the discrete shearlet transform is suited as regularizer for the segmentation of curved structures. Neither the shearlet nor the curvelet transform where used as regularizer in a segmentation model so far. To this end, we have implemented a translation invariant finite discrete shearlet transform based on the fast Fourier transform. We describe how the shearlet transform can be incorporated within the multi-label segmentation model and show how to find a minimizer of the corresponding functional by applying an alternating direction method of multipliers. Here, the Parseval frame property of our shearlets comes into play. We demonstrate by numerical examples that the shearlet-regularized model can better segment curved textures than the TV-regularized one and that the method can also cope with regularizers obtained from non-local means.     

Positional error modeling for line simplification based on automatic shape similarity analysis in GIS

Automatic generalization is a process for representing geographical objects with different degrees of detail on a digital map. The positional error for each geographical object is propagated through the process and a generalization error is also introduced by the generalization. Previous research has focused mainly on measuring the generalization error. This paper presents an analytical model for assessing the positional error in the generalized object by considering both error propagation from the original data and the generalization error. The analytical model provides a shape dissimilarity value that indicates the shape difference between the original data with a positional error and its simplified version. This model is able to objectively and automatically determine the applicability of the generalized data for further applications to geographical information system (GIS) problems. It can also deal with a large amount of data in GIS. Therefore, the analytical model presented, which provides a more comprehensive shape measure for assessing positional error in data derived from the generalization, is valuable in the development of automatic generalization.     

A Sufficient Condition for Invalidation of Linear State-Space Systems With Uncertain Time-Varying Parameters

The model validation problem for linear systems with time-varying parameter uncertainty and additive disturbances is addressed. The disturbance are modelled using the window norm, which is a generalization of the ${rm l}_{infty}$-norm and is shown to be attractive for optimal control. An approximation of the nonlinear operator from parameters to output is found based on the FrÉchet derivative. Using this approximation, a sufficient condition for invalidation of a process model is formulated as a linear feasibility problem. In this condition, an upper bound for the approximation error is included. An overhead crane is used as an illustrative example to show that the model validation test is realistic to perform even with large data sets.      

基于Shearlet的双正则化图像压缩采样恢复

针对图像压缩采样中正交小波变换方向有限和单一正则化的问题,提出了一种基于Shearlet的双正则化图像压缩采样恢复算法。该算法用Shearlet作为图像的稀疏表示,用交替最小化对联合正则化模型进行求解。实验结果表明,该算法恢复的图像与单一的全变分正则化方法和小波变换相比有更好的视觉效果,更高的峰值信噪比。     

基于超图表示的低秩属性选择方法用于回归分析*

针对现有的回归模型未考虑特征之间的深层结构,而导致在回归问题上输出不稳定的模型,提出了一种新的属性选择方法。具体地,通过稀疏学习理论中的 L2,1-范数和 L2,p-范数在线性回归模型分别进行样本降噪和属性选择,然后,利用超图结构和低秩约束来分别考虑数据间的局部结构和不同数据间的全局结构,最后结合子空间学习方法来对模型进行微调。经实验证明,在回归分析中该算法较对比算法能取得更好的效果。     

Relaxed sparse eigenvalue conditions for sparse estimation via non-convex regularized regression

Non-convex regularizers usually improve the performance of sparse estimation in practice. To prove this fact, we study the conditions of sparse estimations for the sharp concave regularizers which are a general family of non-convex regularizers including many existing regularizers. For the global solutions of the regularized regression, our sparse eigenvalue based conditions are weaker than that of L1-regularization for parameter estimation and sparseness estimation. For the approximate global and approximate stationary (AGAS) solutions, almost the same conditions are also enough. We show that the desired AGAS solutions can be obtained by coordinate descent (CD) based methods. Finally, we perform some experiments to show the performance of CD methods on giving AGAS solutions and the degree of weakness of the estimation conditions required by the sharp concave regularizers.