Rapid backpropagation learning algorithms

One of the major drawbacks of the backpropagation algorithm is its slow rate of convergence. Researchers have tried several different approaches to speed up the convergence of backpropagation learning. In this paper, we present those rapid learning methods as three categories, and implement the representative methods of each category: (1) for the numerical method based approach, the Aitken's ² process, (2) for the heuristics based approach, the dynamic adaptation of learning rate, and (3) for the learning strategy based approach, the selective presentation of learning samples. Based on these implementations, the performance is evaluated with experiments and the merits and demerits are briefly discussed.This work was supported in part by a grant from the Korea Science and Engineering Foundation (KOSEF) and the Center for Artificial Intelligence Research (CAIR), the Engineering Research Center (ERC) of Excellence Program.     

几种相关匹配算法性能的研究与分析

为了满足相关匹配的要求,在二元纯位相匹配滤波器的基础上,提出了几种改进的匹配滤波器算法,即优化的二元纯位相匹配滤波器算法、振幅编码位相型匹配滤波器算法及振幅补偿匹配滤波器算法,比较了这些算法的性能,分析了在不同相关匹配条件下的优势。     

Multiple-instance learning algorithms for computer-aided detection

Many computer-aided diagnosis (CAD) problems can be best modelled as a multiple-instance learning (MIL) problem with unbalanced data, i.e., the training data typically consists of a few positive bags, and a very large number of negative instances. Existing MIL algorithms are much too computationally expensive for these datasets. We describe CH, a framework for learning a Convex Hull representation of multiple instances that is significantly faster than existing MIL algorithms. Our CH framework applies to any standard hyperplane-based learning algorithm, and for some algorithms, is guaranteed to find the global optimal solution. Experimental studies on two different CAD applications further demonstrate that the proposed algorithm significantly improves diagnostic accuracy when compared to both MIL and traditional classifiers. Although not designed for standard MIL problems (which have both positive and negative bags and relatively balanced datasets), comparisons against other MIL methods on benchmark problems also indicate that the proposed method is competitive with the state-of-the-art.     

Competitive learning algorithms for robust vector quantization

The efficient representation and encoding of signals with limited resources, e.g., finite storage capacity and restricted transmission bandwidth, is a fundamental problem in technical as well as biological information processing systems. Typically, under realistic circumstances, the encoding and communication of messages has to deal with different sources of noise and disturbances. We propose a unifying approach to data compression by robust vector quantization, which explicitly deals with channel noise, bandwidth limitations, and random elimination of prototypes. The resulting algorithm is able to limit the detrimental effect of noise in a very general communication scenario. In addition, the presented model allows us to derive a novel competitive neural networks algorithm, which covers topology preserving feature maps, the so-called neural-gas algorithm, and the maximum entropy soft-max rule as special cases. Furthermore, continuation methods based on these noise models improve the codebook design by reducing the sensitivity to local minima. We show an exemplary application of the novel robust vector quantization algorithm to image compression for a teleconferencing system     

Intelligent learning algorithms for adaptive digital filters

It is wellknown that gradient search fails in adaptive IIR filters, since their mean-square error surfaces may be multi-modal. In the letter a new approach based on learning algorithms is shown to be capable of performing global optimisation. The new algorithms are suitable for both adaptive FIR and IIR filters.<>     

Stochastic approximation algorithms for partition functionestimation of Gibbs random fields

We present an analysis of previously proposed Monte Carlo algorithms for estimating the partition function of a Gibbs random field. We show that this problem reduces to estimating one or more expectations of suitable functionals of the Gibbs states with respect to properly chosen Gibbs distributions. As expected, the resulting estimators are consistent. Certain generalizations are also provided. We study computational complexity with respect to grid size and show that Monte Carlo partition function estimation algorithms can be classified into two categories: E-type algorithms that are of exponential complexity and P-type algorithms that are of polynomial complexity, Turing reducible to the problem of sampling from the Gibbs distribution. E-type algorithms require estimating a single expectation, whereas, P-type algorithms require estimating a number of expectations with respect to Gibbs distributions which are chosen to be sufficiently “close” to each other. In the latter case, the required number of expectations is of polynomial order with respect to grid size. We compare computational complexity by using both theoretical results and simulation experiments. We determine the most efficient E-type and P-type algorithms and conclude that P-type algorithms are more appropriate for partition function estimation. We finally suggest a practical and efficient P-type algorithm for this task     

Iterative learning algorithms for linear Gaussian observation models

In this paper, we consider a signal/parameter estimation problem that is based on a linear model structure and a given setting of statistical models with unknown hyperparameters. We consider several combinations of Gaussian and Laplacian models. We develop iterative algorithms based on two typical machine learning methods - the evidence-based method and the integration-based method - to deal with the hyperparameters. We have applied the proposed algorithms to adaptive prediction and wavelet denoising. In linear prediction, we show that the proposed algorithms are efficient tools for tackling a difficult problem of adapting simultaneously the order and the coefficients of the predictor. In wavelet denoising, we show that by using the proposed algorithms, the noisy wavelet coefficients are subject to shrinkage and thresholding.     

距离度量学习算法的精确性研究

距离度量学习是机器学习领域较为活跃的研究课题之一,文中利用UCI(加州大学欧文分校)数据库的数据对度量学习算法进行比较研究。为了寻找一种可靠的没有明确定义标志的算法,选择四种算法在UCI的六个数据集上对距离矩阵进行比较。每个样本数据集的性质(尺寸和维度)是不同的,因此算法的结果也不同。编码相似度算法在大多数情况下表现良好。在未来的实际应用领域,对于提高无标记数据和相似集的距离度量学习算法的精确性提供了研究基础。     

On the generalization ability of on-line learning algorithms

In this paper, it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on-line learning algorithm run on an independent and identically distributed (i.i.d.) sample of data. Using a simple large deviation argument, we prove tight data-dependent bounds for the risk of this hypothesis in terms of an easily computable statistic M/sub n/ associated with the on-line performance of the ensemble. Via sharp pointwise bounds on M/sub n/, we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix. These bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions, an issue that was left open by previous results. A distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences; i.e., a model making no probabilistic assumptions on the source generating the data. In fact, these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds.     

基于不变量的机器学习算法分析

随着机器学习算法在人工智能各领域的广泛应用,人们开始关注机器学习算法的质量分析.由于机器学习算法中缺少测试语言,对学习算法进行质量分析是很困难的,基于此,本文提出了基于不变量的机器学习算法分析方法,对5种机器学习算法进行不同参数下的不变量生成,得到不变量集合.通过动态筛选机制和函数调用图,对不变量集合进行筛选和提取,得...     

深度学习在影像中的算法研究

深度学习技术的运用正日趋广泛,深度学习自身的高效性和智能性受到研究者的青睐.通过对深度学习影像分类的剖析,进一步探究深度学习在影像识别方向的应用,介绍了主要用于影像分类识别的基于深度学习的医疗影像检测算法,可作为开展深度学习技术运用于医学影像检测研究工作的有益参考.     

Application of constructive learning algorithms to the inverseproblem

A constructive learning algorithm is used to generate networks that learn to approximate the functional of the magnetotelluric inverse problem. Based on synthetic data, several experiments are performed in order to generate and test the neural networks. Rather than producing, at the present time, a practical algorithm using this approach, the object of the paper is to explore the possibilities offered by the new tools. The generated networks can be used as an internal module in a more general inversion program, or their predicted models can be used by themselves or simply as inputs to an optimization program     

New algorithms for learning and pruning oblique decision trees

We present methods for learning and pruning oblique decision trees. We propose a new function for evaluating different split rules at each node while growing the decision tree. Unlike the other evaluation functions currently used in the literature (which are all based on some notion of purity of a node), this new evaluation function is based on the concept of degree of linear separability. We adopt a correlation based optimization technique called the Alopex algorithm (K.P. Unnikrishnaan and K.P. Venugopal, 1994) for finding the split rule that optimizes our evaluation function at each node. The algorithm we present is applicable only for 2-class problems. Through empirical studies, we demonstrate that our algorithm learns good compact decision trees. We suggest a representation scheme for oblique decision trees that makes explicit the fact that an oblique decision tree represents each class as a union of convex sets bounded by hyperplanes in the feature space. Using this representation, we present a new pruning technique. Unlike other pruning techniques, which generally replace heuristically selected subtrees of the original tree by leaves, our method can radically restructure the decision tree. Through empirical investigation, we demonstrate the effectiveness of our method     

太赫兹光谱结合机器学习的甜味剂检测

以三氯蔗糖、赤藓糖醇、木糖醇3种人工甜味剂为研究对象,采用太赫兹时域光谱技术,结合多种机器学习和优化算法对甜味剂与面粉混合物的光谱数据进行系统的分类识别和定量回归研究。结果表明,麻雀搜索算法-支持向量机模型/支持向量回归模型(SSA-SVM/SVR)对混合物的定性及定量分析结果均达到最优,分类预测的准确率达到95.56%,定量回归预测的最佳回归系数R2为0.999 8,实现了3种甜味剂和面粉混合物的高精确度分类和定量分析,为人工甜味剂的快速检测提供了一种有效可靠的新思路。     

基于深度学习的图像去雾算法研究进展

图像是信息的重要承载形式。雾霾的出现降低了图像采集设备采集到的图像质量,容易出现色彩暗淡、对比度和饱和度降低、细节信息丢失等问题,直接影响了有用信息的表达和利用。目前对图像去雾的研究多采用深度学习的方法,卷积神经网络代替了人工特征提取方式,取得了优于传统算法的去雾效果,但普遍存在着对真实世界雾霾图像和清晰图像对的依赖。无监督学习的方法带来了新的解决思路。从监督学习和无监督学习的角度对有代表性的深度学习图像去雾算法进行分类,归纳了常用的数据集、评价指标,概括分析了有影响力的去雾模型的核心思想,总结了各算法的优缺点和适用场景。针对目前工作存在的不足,探索了下一步研究的方向。     

基于深度学习的异构资源分配算法研究

资源分配是目前云计算领域中一个重要的研究方向。在异构云计算体系结构下的复杂应用问题研究中,为了满足异构资源分配的需求,提升资源利用效率,文中提出了一种基于深度学习的面向应用的资源分配算法。该算法将数据特征进行量化,更加精确地刻画了不同服务器资源之间的性能差异,在分配算法中加入了一个工作负载预测模型,使给出的资源分配方案与需求更加匹配,同时提高了资源利用率。     

基于流形学习的新高光谱图像降维算法

提出了一种新的基于图像块距离的邻域选择方法,并将其应用于流形学习中,得到一类新的高光谱图像非线性降维算法。该类算法利用高光谱图像物理特性,结合图像的光谱信息和空间信息,在最大限度减小图像信息冗余的基础之上,很好地保持了原始数据集的特性。与其它高光谱图像的降维算法相比,改进的流形学习算法不仅考虑到高光谱图像本身的空间关系,而且利用图像块距离更好地保持了数据点之间的局部特性,从而有效地去除原始数据集光谱维和空间维的冗余信息。实际高光谱数据的实验结果表明,所提出的算法在应用于高光谱图像分类时,与其它方法相比具有更高的分类精度。     

Neural network constructive algorithms: Trading generalization for learning efficiency?

There are currently several types of constructive, (or growth), algorithms available for training a feed-forward neural network. This paper describes and explains the main ones, using a fundamental approach to the multi-layer perceptron problem-solving mechanisms. The claimed convergence properties of the algorithms are verified using just two mapping theorems, which consequently enables all the algorithms to be unified under a basic mechanism. The algorithms are compared and contrasted and the deficiencies of some highlighted. The fundamental reasons for the actual success of these algorithms are extracted, and used to suggest where they might most fruitfully be applied. A suspicion that they are not a panacea for all current neural network difficulties, and that one must somewhere along the line pay for the learning efficiency they promise, is developed into an argument that their generalization abilities will lie on average below that of back-propagation.Funded by the German Ministry of Research and Technology, grant number 01 IN 111 A/4.German National Research Centre for Computer Science (GMD), Schloß Birlinghoven, 5205 St. Augustin 1, Germany.     

Stochastic gradient algorithms for design of minimum error-rate linear dispersion codes in MIMO wireless systems

Linear dispersion (LD) codes are a good candidate for high-data-rate multiple-input multiple-ouput (MIMO) signaling. Traditionally LD codes were designed by maximizing the average mutual information, which cannot guarantee good error performance. This paper presents a new design scheme for LD codes that directly minimizes the block error rate (BLER) in MIMO channels with arbitrary fading statistics and various detection algorithms. For MIMO systems employing LD codes, the error rate does not admit an explicit form. Therefore, we cannot use deterministic optimization methods to design the minimum-error-rate LD codes. In this paper, we propose a simulation-based optimization methodology for the design of LD codes through stochastic approximation and simulation-based gradient estimation. The gradient estimation is done using the score function method originally developed in the discrete-event-system community. The proposed method can be applied to design the minimum-error-rate LD codes for a variety of detector structures including the maximum-likelihood (ML) detector and several suboptimal detectors. It can also design optimal codes under arbitrary fading channel statistics; in particular, it can take into account the knowledge of spatial fading correlation at the transmitter and receiver ends. Simulation results show that codes generated by the proposed new design paradigm generally outperform the codes designed based on algebraic number theory.     

Segmentation of magnetic resonance images using fuzzy algorithms for learning vector quantization

This paper evaluates a segmentation technique for magnetic resonance (MR) images of the brain based on fuzzy algorithms for learning vector quantization (FALVQ). These algorithms perform vector quantization by updating all prototypes of a competitive network through an unsupervised learning process. Segmentation of MR images is formulated as an unsupervised vector quantization process, where the local values of different relaxation parameters form the feature vectors which are represented by a relatively small set of prototypes. The experiments evaluate a variety of FALVQ algorithms in terms of their ability to identify different tissues and discriminate between normal tissues and abnormalities.