高斯过程回归方法综述

高斯过程回归是基于贝叶斯理论和统计学习理论发展起来的一种全新机器学习方法,适于处理高维数、小样本和非线性等复杂回归问题。在阐述该方法原理的基础上,分析了其存在的计算量大、噪声必须服从高斯分布等问题,给出了改进方法。与神经网络和支持向量机相比,该方法具有容易实现、超参数自适应获取以及输出具有概率意义等优点,方便与预测控制、自适应控制、贝叶斯滤波等相结合。最后总结了其应用情况并展望了未来发展方向。     

Ranking importance of input parameters of neural networks

Artificial neural networks have been used for simulation, modeling, and control purposes in many engineering applications as an alternative to conventional expert systems. Although neural networks usually do not reach the level of performance exhibited by expert systems, they do enjoy a tremendous advantage of very low construction costs. This paper addresses the issue of identifying important input parameters in building a multilayer, backpropagation network for a typical class of engineering problems. These problems are characterized by having a large number of input variables of varying degrees of importance; and identifying the important variables is a common issue since elimination of the unimportant inputs leads to a simplification of the problem and often a more accurate modeling or solution. We compare three different methods for ranking input importance: sensitivity analysis, fuzzy curves, and change of MSE (mean square error); and analyze their effectiveness. Simulation results based on experiments with simple mathematical functions as well as a real engineering problem are reported. Based on the analysis and our experience in building neural networks, we also propose a general methodology for building backpropagation networks for typical engineering applications.     

Feature and decision level sensor fusion of electromagnetic induction and ground penetrating radar sensors for landmine detection with hand-held units

Strategies for fusion of electromagnetic induction (metal detector (MD)) and ground penetrating radar (GPR) sensors for landmine detection are investigated. Feature and decision level algorithms are devised and compared. Features are extracted from the MD signals by correlating with weighted density distribution functions. A multi-frequency band linear prediction method generates features for the GPR. Feature level fusion combines MD and GPR features in a single neural network. Decision level fusion is performed by using the MD features as inputs to one neural network and the GPR features as inputs to the geometric mean and combining the output values. Experimental results are reported on a very large real data set containing 2315 mine encounters of different size, shape, content and metal composition that are measured under different soil conditions at three distinct geographical locations.     

基于多标签层次聚类的GPR 图像双曲波提取方法

探地雷达(GPR)图像双曲波提取是分析地下目标位置和结构的重要方法,但在真 实环境中,由于噪声和杂波的干扰,使得提取出的双曲波存在结构不完整、碎片化和形状异 常等问题,不利于数据分析和三维建模等后续操作。为此,提出了一种基于多标签层次聚类 的双曲波提取方法(MHCE)。首先通过信息熵评价像素邻域的稳定性,构造了基于信息熵的 距离度量来进行层次聚类;然后利用聚类后的邻接空间进行多标签聚类以降低杂波和噪声对 双曲波提取的影响;最后结合多标签聚类结果的拟合形状和纹理方向提取双曲波。实验表明, 该方法对于真实GPR 图像双曲波具有较好的鲁棒性,能够获得规范化的双曲波形状和位置 参数。     

Combining deterministic modelling with artificial neural networks for suspended sediment estimates

Estimates of suspended sediment concentrations and transport are an important part of any marine environment assessment study because these factors have a direct impact on the life cycle and survival of marine ecosystems. This paper proposes to implement a combined methodology to tackle these estimates. The first component of the methodology comprised two numerical current and wave models, while the second component was based on the artificial intelligence technique of neural networks (ANNs) used to reproduce values of sediment concentrations observed at two sites. The ANNs were fed with modelled currents and waves and trained to produce area-specific concentration estimates. The trained ANNs were then applied to predict sediment concentrations over an independent period of observations. The use of a data set that merged together observations from both the mentioned sites provided the best ANN testing results in terms of both the normalised root mean square error (0.13) and the mean relative error (0.02).     

A sparse Gaussian process regression model for tourism demand forecasting in Hong Kong

In recent years, Gaussian process (GP) models have been popularly studied to solve hard machine learning problems. The models are important due to their flexible non-parametric modeling abilities using Mercer kernels and the Bayesian framework for probabilistic inference. In this paper, we propose a sparse GP regression (GPR) model for tourism demand forecasting in Hong Kong. The sparsification procedure of the GPR model not only decreases the computational complexity but also improves the generalization ability. We experiment the proposed model with monthly demand data that are relevant to Hong Kong’s tourism industry, and compare the performance of the sparse GPR model with those of various kernel-based models to show its effectiveness. The proposed sparse GPR model shows that its forecasting capability outperforms those of the ARMA model and the two state-of-the-art SVM models.     

GPR target detection using a neural network classifier designed by a multi-objective genetic algorithm

Ground Penetrating Radar (GPR) is an electromagnetic sensing technology employed for localization of underground utilities, pipes, and other types of objects. The radargrams typically obtained have a high dimensionality, containing a number of signatures with hyperbolic pattern shapes, and can be processed to retrieve information about the target’s locations, depths and material type of underground soil. The classical Hough Transform approach used to reconstruct these hyperbola shapes is computationally expensive, given the large dimensionality of the radargrams. In literature, several approaches propose to first approximate the location of hyperbolas to small segments through a classification stage, before applying the Hough transform over these segments. However, the published classifiers designed for this task present a relatively complex architecture.Aiming at an improved target localization, we propose an alternative classification methodology. The goal is to classify windows of GPR radargrams into two classes (with or without target) using a neural network radial basis function (RBF), designed via a multi-objective genetic algorithm (MOGA). To capture samples’ fine details, high order statistic cumulant features (HOS) were used. Feature selection was performed by MOGA, with an optional prior reduction using a mutual information (MIFS) approach. The obtained results demonstrate improvement of the classification performance when compared with other models designed with the same data and are among the best results available in the literature, albeit the large reduction in classifier complexity.     

一种改进的前馈神经网络结构及其在复杂系统建模中的应用

针对包含多道加工工序、输入变量很多的复杂工业系统建模精度难以提高的问题,提出一种改进的前馈神经网络结构,输入变量不是由同一层输入,而是根据变量起作用的前后次序分别在网络的不同层输入,真实反映了大工业过程的各生产工序中的参数发生作用的时间顺序。同时由于输入变量在适当的时候输入网络,从而使网络的规模减小。该神经网络是处理高维问题,尤其是建立包含多道加工工序的大工业过程模型问题的强有力工具。将该神经网络用于热连轧产品质量建模,经过实测数据拟合与检验,仿真结果表明:提出的小波神经网络结构是可行的而且有很好的应用前景。     

Design of ANN Based Non-Linear Network Using Interconnection of Parallel Processor

Suspicious mass traffic constantly evolves, making network behaviour tracing and structure more complex. Neural networks yield promising results by considering a sufficient number of processing elements with strong interconnections between them. They offer efficient computational Hopfield neural networks models and optimization constraints used by undergoing a good amount of parallelism to yield optimal results. Artificial neural network (ANN) offers optimal solutions in classifying and clustering the various reels of data, and the results obtained purely depend on identifying a problem. In this research work, the design of optimized applications is presented in an organized manner. In addition, this research work examines theoretical approaches to achieving optimized results using ANN. It mainly focuses on designing rules. The optimizing design approach of neural networks analyzes the internal process of the neural networks. Practices in developing the network are based on the interconnections among the hidden nodes and their learning parameters. The methodology is proven best for nonlinear resource allocation problems with a suitable design and complex issues. The ANN proposed here considers more or less 46k nodes hidden inside 49 million connections employed on full-fledged parallel processors. The proposed ANN offered optimal results in real-world application problems, and the results were obtained using MATLAB.     

A general methodology for designing globally convergentoptimization neural networks

We present a general methodology for designing optimization neural networks. We prove that the neural networks constructed by using the proposed method are guaranteed to be globally convergent to solutions of problems with bounded or unbounded solution sets, in contrast with the gradient methods whose convergence is not guaranteed. We show that the proposed method contains both the gradient methods and nongradient methods employed in existing optimization neural networks as special cases. Based on the theoretical results of the proposed method, we study the convergence and stability of general gradient models in the case of unisolated solutions. Using the proposed method, we derive some new neural network models for a very large class of optimization problems, in which the equilibrium points correspond to exact solutions and there is no variable parameter. Finally, some numerical examples show the effectiveness of the method.     

A Policy Evaluation Tool for Multisite Resource Management

An enterprise typically operates multiple datacenter sites, each handling workloads according to an enterprise-level strategy. Sharing resources across multiple sites (or enterprises) brings up several important problems. Each site may have its own policies that govern its interactions with other remote sites. Different policies impact the system performance in different ways. The site administrators and system designers need to understand the effects of a given set of policies on different workloads. In this paper, we describe an analysis methodology that determines the impact of policies on the workloads, and we present results and validation for a prototypical multi-site resource sharing system. Our analytical tool is capable of evaluating complex policies on a large scale system and permits independent policies for each site, so that policy makers can quickly evaluate several alternatives and their effects on the workloads before deploying them.     

一种基于输入训练神经网络的非线性PCA 故障诊断方法

简要讨论了线性PCA故障诊断方法存在的问题，提出一种基于输入训练神经网络的非线性PCA故障诊断方法。该方法首先利用输入训练神经网络和BP网络双网络机制，实现非线性主元的识别，并采用统计方法进行故障检测与故障分离。对CSTR的仿真研究结果表明，该方法能够克服线性PCA方法在提取过程变量的非线性特征方面存在的不足，并能够准确地进行故障检测和分离。     

Preliminary study on wilcoxon learning machines.

As is well known in statistics, the resulting linear regressors by using the rank-based Wilcoxon approach to linear regression problems are usually robust against (or insensitive to) outliers. This motivates us to introduce in this paper the Wilcoxon approach to the area of machine learning. Specifically, we investigate four new learning machines, namely Wilcoxon neural network (WNN), Wilcoxon generalized radial basis function network (WGRBFN), Wilcoxon fuzzy neural network (WFNN), and kernel-based Wilcoxon regressor (KWR). These provide alternative learning machines when faced with general nonlinear learning problems. Simple weights updating rules based on gradient descent will be derived. Some numerical examples will be provided to compare the robustness against outliers for various learning machines. Simulation results show that the Wilcoxon learning machines proposed in this paper have good robustness against outliers. We firmly believe that the Wilcoxon approach will provide a promising methodology for many machine learning problems.     

Predicting sea-level variations at the Cocos (Keeling) Islands with artificial neural networks

Sea-level variations affect the construction and management of coastal structures, near-shore navigation, coastal rivers’ hydrological regime, and coastal tourism. Estimates of sea-level with hours-to-days warning times are especially important for low-lying regions, such as the Cocos (Keeling) Islands in the Indian Ocean. This study employs the technique of artificial neural networks to predict sea-level variations with warning times from 1 h to 5 days on the basis of hourly tide gauge observations. The data from the Cocos (Keeling) Islands SEAFRAME tide station for the period from 1992 to 2003 were used here. Feed-forward three-layered artificial neural networks were implemented to simulate sea level. The proposed neural methodology demonstrated reliable results in terms of the correlation coefficient (0.85–0.95), root mean square error (80–100 mm), and scatter index (0.1–0.2) when compared with actual observations. Therefore, the proposed methodology could be successfully used for site-specific forecasts.     

Logistics planning under uncertainty for disposition of radioactive wastes

The US Department of Energy (DOE) faces an enormous environmental remediation challenge involving highly radioactive wastes at former weapons production facilities. The purpose of this analysis is to focus on equipment acquisition and fleet sizing issues related to transportation of wastes from remediation sites to disposal sites. Planning for the transportation of these wastes must be done with recognition of important uncertainties related to overall quantities of waste to be moved, the rate at which the wastes will be prepared for transport, and the certification of suitable transportation containers for use in the effort. However, deadlines for completion of the effort have already been set by the political process, without much regard for these uncertainties. To address this fleet sizing problem, we have created a robust optimization model that focuses on equipment investment decisions. Through this robust optimization, we illustrate how modeling can be used to explore the effects of uncertainty on the equipment acquisition strategy. The disposition of radioactive wastes from DOE sites is an important illustration of a category of problems where equipment investments must be made under conditions of considerable uncertainty. The methodology illustrated in this paper can be applied to this general class of problems.     

Landmine detection and classification with complex-valued hybrid neural network using scattering parameters dataset

Neural networks have been applied to landmine detection from data generated by different kinds of sensors. Real-valued neural networks have been used for detecting landmines from scattering parameters measured by ground penetrating radar (GPR) after disregarding phase information. This paper presents results using complex-valued neural networks, capable of phase-sensitive detection followed by classification. A two-layer hybrid neural network structure incorporating both supervised and unsupervised learning is proposed to detect and then classify the types of landmines. Tests are also reported on a benchmark data.     

Polynomial-based radial basis function neural networks (P-RBF NNs) and their application to pattern classification

Polynomial neural networks have been known to exhibit useful properties as classifiers and universal approximators. In this study, we introduce a concept of polynomial-based radial basis function neural networks (P-RBF NNs), present a design methodology and show the use of the networks in classification problems. From the conceptual standpoint, the classifiers of this form can be expressed as a collection of “if-then” rules. The proposed architecture uses two essential development mechanisms. Fuzzy clustering (Fuzzy C-Means, FCM) is aimed at the development of condition parts of the rules while the corresponding conclusions of the rules are formed by some polynomials. A detailed learning algorithm for the P-RBF NNs is developed. The proposed classifier is applied to two-class pattern classification problems. The performance of this classifier is contrasted with the results produced by the “standard” RBF neural networks. In addition, the experimental application covers a comparative analysis including several previous commonly encountered methods such as standard neural networks, SVM, SOM, PCA, LDA, C4.5, and decision trees. The experimental results reveal that the proposed approach comes with a simpler structure of the classifier and better prediction capabilities.     

带有协方差矩阵的卷积神经网络在人体运动识别中的应用

目前,深度学习已经在各种人体运动识别(HAR)任务中发挥了重要作用。但是,由于运动数据具有时间序列和包含肢体动作的特殊性,现有神经网络在进行卷积操作时会导致数据高度相关,并且随着网络影响到下一层,这限制了模型的识别效果。为此,提出了一种带有协方差矩阵的改进卷积神经网络用于HAR场景,通过矩阵变换搭建一种去相关的网络结构来消除相关性问题,可以在网络表现不佳时替代现有的批量归一化(BN)层用于归一化数据。在4个HAR公共数据集上进行实验,并与传统CNN和带有BN层的模型进行比较。实验结果表明,对比此前的深度学习网络,改进的神经网络有1%～2%的性能提升,验证了该方法的有效性,并将程序移植到了移动端进行实时运动识别。     

Artificial neural networks compared to factor analysis for low-dimensional classification of high-dimensional body fat topography data of healthy and diabetic subjects.

Subcutaneous adipose tissue thickness was measured in 590 healthy subjects at 15 specific body sites by means of the new optical device, lipometer, providing a high-dimensional and partly highly intercorrelated set of data, which had been analyzed by factor analysis previously. N-2-N back-propagation neural networks are able to perform low-dimensional display of high-dimensional data as a special application. We report about the performance of such a 15-2-15 network and compare its results with the output of factor analysis. As test data for verification, measurement values on women with proven diabetes mellitus type II (NIDDM) are used. Surprisingly our 15-2-15 neural network is able to reproduce the classification pattern resulting from factor analysis very precisely. After extracting the network weights the classification of new subjects is even more simple with the neural network as compared with factor analysis. In addition, the network weights are able to cluster highly correlated body sites nicely to different groups, corresponding to different regions of the human body. Thus, the analysis of these weights provides additional information about the structure of the data. Therefore, N-2-N networks seem to be a good alternative method for analyzing high-dimensional data with strong intercorrelation.     

Machine learning regression algorithms for biophysical parameter retrieval: Opportunities for Sentinel-2 and -3

ESA's upcoming satellites Sentinel-2 (S2) and Sentinel-3 (S3) aim to ensure continuity for Landsat 5/7, SPOT-5, SPOT-Vegetation and Envisat MERIS observations by providing superspectral images of high spatial and temporal resolution. S2 and S3 will deliver near real-time operational products with a high accuracy for land monitoring. This unprecedented data availability leads to an urgent need for developing robust and accurate retrieval methods. Machine learning regression algorithms may be powerful candidates for the estimation of biophysical parameters from satellite reflectance measurements because of their ability to perform adaptive, nonlinear data fitting.By using data from the ESA-led field campaign SPARC (Barrax, Spain) we have compared the utility of four state-of-the-art machine learning regression algorithms and four different S2 and S3 band settings to assess three important biophysical parameters: leaf chlorophyll content (Chl), leaf area index (LAI) and fractional vegetation cover (FVC). The tested Sentinel configurations were: S2-10 m (4 bands), S2-20 m (8 bands), S2-60 m (10 bands) and S3-300 m (19 bands), and the tested methods were: neural networks (NN), support vector regression (SVR), kernel ridge regression (KRR), and Gaussian processes regression (GPR).GPR outperformed the other retrieval methods for the majority of tested configurations and was the only method that reached the 10% precision required by end users in the estimation of Chl. Also, although validated with an RMSE accuracy around 20%, GPR yielded optimal LAI and FVC estimates at highest S2 spatial resolution of 10 m with only four bands. In addition to high accuracy values, GPR also provided confidence intervals of the estimates and insight in relevant bands, which are key advantages over the other methods. Given all this, GPR proved to be a fast and accurate nonlinear retrieval algorithm that can be potentially implemented for operational monitoring applications.