Explicit Fourth-Order Runge–Kutta Method on Intel Xeon Phi Coprocessor

This paper concerns an Intel Xeon Phi implementation of the explicit fourth-order Runge–Kutta method (RK4) for very sparse matrices with very short rows. Such matrices arise during Markovian modeling of computer and telecommunication networks. In this work an implementation based on Intel Math Kernel Library (Intel MKL) routines and the authors’ own implementation, both using the CSR storage scheme and working on Intel Xeon Phi, were investigated. The implementation based on the Intel MKL library uses the high-performance BLAS and Sparse BLAS routines. In our application we focus on OpenMP style programming. We implement SpMV operation and vector addition using the basic optimizing techniques and the vectorization. We evaluate our approach in native and offload modes for various number of cores and thread allocation affinities. Both implementations (based on Intel MKL and made by the authors) were compared in respect of the time, the speedup and the performance. The numerical experiments on Intel Xeon Phi show that the performance of authors’ implementation is very promising and gives a gain of up to two times compared to the multithreaded implementation (based on Intel MKL) running on CPU (Intel Xeon processor) and even three times in comparison with the application which uses Intel MKL on Intel Xeon Phi.     

基于改进KPCA与混合核函数LSSVR的泥石流预测

针对引发泥石流灾害的多重影响因素而导致的预测维数灾难，以及最小二乘支持向量回归（least squares support vector regression， LSSVR）模型中选取单核函数而导致的模型训练性能部分缺陷的问题，提出了一种基于改进的核主成分分析（kernel principal component analysis， KPCA）与混合核函数LSSVR的泥石流灾害预测方法.首先，将影响泥石流发生的7种初始因子赋予权重，利用加权KPCA法筛选出3个主成分影响因子作为模型输入；然后，将局部核函数与全局核函数相结合，运用到LSSVR模型上，进行泥石流发生概率预测，以平衡样本学习能力与泛化能力，并使用果蝇优化算法（fruit fly optimization algorithm， FOA）更新模型的最优参数；最后，以磨子沟监测数据进行仿真验证.结果表明，该方法能够有效地降低维数灾难并提升预测模型精确度，在误差允许范围内预测出泥石流发生概率值及对应的预警等级，为相关决策部门提供一定的借鉴经验.     

Hybrid Recommendation System for Heart Disease Diagnosis based on Multiple Kernel Learning with Adaptive Neuro-Fuzzy Inference System

Multiple Kernel Learning with Adaptive Neuro-Fuzzy Inference System (MKL with ANFIS) based deep learning method is proposed in this paper for heart disease diagnosis. The proposed MKL with ANFIS based deep learning method follows two-fold approach. MKL method is used to divide parameters between heart disease patients and normal individuals. The result obtained from the MKL method is given to the ANFIS classifier to classify the heart disease and healthy patients. Sensitivity, Specificity and Mean Square Error (MSE) are calculated to evaluate the proposed MKL with ANFIS method. The proposed MKL with ANFIS is also compared with various existing deep learning methods such as Least Square with Support Vector Machine (LS with SVM), General Discriminant Analysis and Least Square Support Vector Machine (GDA with LS-SVM), Principal Component Analysis with Adaptive Neuro-Fuzzy Inference System (PCA with ANFIS) and Latent Dirichlet Allocation with Adaptive Neuro-Fuzzy Inference System (LDA with ANFIS). The results from the proposed MKL with ANFIS method has produced high sensitivity (98%), high specificity (99%) and less Mean Square Error (0.01) for the for the KEGG Metabolic Reaction Network dataset.     

A computationally efficient multi-domain active learning method for crop mapping using satellite image time-series

Crop mapping through classification of Satellite Image Time-Series (SITS) data can provide precious information for several agricultural applications, such as crop monitoring, yield forecasting, and crop inventory. However, several issues affect the classification performance of SITS data. As one of the most challenging problems, constituent images of time-series provide different levels of information about crops. These differences are the result of dynamic spectral responses of crops and also the variable atmospheric and sensor conditions. The second issue is the unavailability of adequate high-quality samples for training the classifier. In this study, we proposed a novel computationally efficient Multi-Domain Active Learning (MDAL) method which takes advantage of Multiple Kernel Learning (MKL) and Active Learning (AL) algorithms to address these two issues. The proposed method uses MKL algorithms to address the issues associated with different information level of the data, which generally cannot be modelled using the well-known classification algorithms. AL algorithms were also used for semi-automatic selection of training samples. However, most of the MKL algorithms are very computationally demanding. Consequently, using them in the MDAL method can dramatically increase the computational costs. Thus, in this paper, we presented the similarity-based MKL algorithms. Thanks to their low computational complexities, these algorithms are the most suitable MKL algorithms that can be used in the MDAL method. We evaluated the proposed method using two multispectral SITS datasets, acquired by RapidEye and SPOT sensors. The obtained results of the MDAL method for these datasets respectively showed 8.2% and 5.87% increase in the overall accuracy of classification as compared to the accuracy of the standard AL algorithm. The results also showed that in the case of adopting the SimpleMKL algorithm (a common MKL algorithm in the literature) the computational time of the MDAL method is 577 and 474 seconds for RapidEye and SPOT datasets, respectively. However, in the case of adopting the similarity-based MKL algorithms, these computational times respectively decreases to 4 and 2 seconds.     

通用稀疏多核学习

针对L1范数多核学习方法产生核权重的稀疏解时可能会导致有用信息的丢失和泛化性能退化,Lp范数多核学习方法产生核权重的非稀疏解时会产生很多冗余信息并对噪声敏感,提出了一种通用稀疏多核学习方法。该算法是基于L1范数和Lp范数(p>1) 混合的网状正则化多核学习方法,不仅能灵活的调整稀疏性,而且鼓励核权重的组效应,L1范数和Lp范数多核学习方法可以认为是该方法的特例。该方法引进的混合约束为非线性约束,故对此约束采用二阶泰勒展开式近似,并使用半无限规划来求解该优化问题。实验结果表明,改进后的方法在动态调整稀疏性的前提下能获得较好的分类性能,同时也支持组效应,从而验证了改进后的方法是有效可行的。     

Image category learning and classification via optimal linear combination of multiple partially matching kernels

Multiple kernel learning (MKL) aims at simultaneously optimizing kernel weights while training the support vector machine (SVM) to get satisfactory classification or regression results. Recent publications and developments based on SVM have shown that by using MKL one can enhance interpretability of the decision function and improve classifier performance, which motivates researchers to explore the use of homogeneous model obtained as linear combination of various types of kernels. In this paper, we show that MKL problems can be solved efficiently by modified projection gradient method and applied for image categorization and object detection. The kernel is defined as a linear combination of feature histogram function that can measure the degree of similarity of partial correspondence between feature sets for discriminative classification, which allows recognition robust to within-class variation, pose changes, and articulation. We evaluate our proposed framework on the ETH-80 dataset for several multi-level image encodings for supervised and unsupervised object recognition and report competitive results.     

在线鲁棒最小二乘支持向量机回归建模

鉴于工业过程的时变特性以及现场采集的数据通常具有非线性特性且包含离群点,利用最小二乘支持向量机回归（least squares support vector regression,LSSVR）建模易受离群点的影响.针对这一问题,结合鲁棒学习算法（robust learning algorithm,RLA）,本文提出了一种在线鲁棒最小二乘支持向量机回归建模方法.该方法首先利用LSSVR模型对过程输出进行预测,与真实输出相比较得到预测误差;然后利用RLA方法训练LSSVR模型的权值,建立鲁棒LSSVR模型;最后应用增量学习方法在线更新鲁棒LSSVR模型,从而得到在线鲁棒LSSVR模型.仿真研究验证了所提方法的有效性.     

Analytic center cutting plane method for multiple kernel learning

Multiple Kernel Learning (MKL) is a popular generalization of kernel methods which allows the practitioner to optimize over convex combinations of kernels. We observe that many recent MKL solutions can be cast in the framework of oracle based optimization, and show that they vary in terms of query point generation. The popularity of such methods is because the oracle can fortuitously be implemented as a support vector machine. Motivated by the success of centering approaches in interior point methods, we propose a new approach to optimize the MKL objective based on the analytic center cutting plane method (accpm). Our experimental results show that accpm outperforms state of the art in terms of rate of convergence and robustness. Further analysis sheds some light as to why MKL may not always improve classification accuracy over naive solutions.     

基于牛顿梯度优化的弹性多核学习

已有稀疏多核学习(MKL)模型在产生核函数权重稀疏解时容易导致信息丢失且泛化能力差,且基于梯度下降法的MKL在接近最优解时收敛速度慢.建立了基于支持向量机(SVM)的弹性多核学习(EMKL)模型并给出了一种基于牛顿梯度优化的EMKL(NO-EMKL).模型在MKL的目标函数中引入弹性项,并设计了基于二阶牛顿梯度下降法的优化算法.实验结果表明:算法不仅具有更好的分类精度,还具有较快的收敛速度.     

Non-uniform multiple kernel learning with cluster-based gating functions

Recently, multiple kernel learning (MKL) has gained increasing attention due to its empirical superiority over traditional single kernel based methods. However, most of state-of-the-art MKL methods are “uniform” in the sense that the relative weights of kernels keep fixed among all data.Here we propose a “non-uniform” MKL method with a data-dependent gating mechanism, i.e., adaptively determine the kernel weights for the samples. We utilize a soft clustering algorithm and then tune the weight for each cluster under the graph embedding (GE) framework. The idea of exploiting cluster structures is based on the observation that data from the same cluster tend to perform consistently, which thus increases the resistance to noises and results in more reliable estimate. Moreover, it is computationally simple to handle out-of-sample data, whose implicit RKHS representations are modulated by the posterior to each cluster.Quantitative studies between the proposed method and some representative MKL methods are conducted on both synthetic and widely used public data sets. The experimental results well validate its superiorities.     

A primal method for multiple kernel learning

The canonical support vector machines (SVMs) are based on a single kernel, recent publications have shown that using multiple kernels instead of a single one can enhance interpretability of the decision function and promote classification accuracy. However, most of existing approaches mainly reformulate the multiple kernel learning as a saddle point optimization problem which concentrates on solving the dual. In this paper, we show that the multiple kernel learning (MKL) problem can be reformulated as a BiConvex optimization and can also be solved in the primal. While the saddle point method still lacks convergence results, our proposed method exhibits strong optimization convergence properties. To solve the MKL problem, a two-stage algorithm that optimizes canonical SVMs and kernel weights alternately is proposed. Since standard Newton and gradient methods are too time-consuming, we employ the truncated-Newton method to optimize the canonical SVMs. The Hessian matrix need not be stored explicitly, and the Newton direction can be computed using several Preconditioned Conjugate Gradient steps on the Hessian operator equation, the algorithm is shown more efficient than the current primal approaches in this MKL setting. Furthermore, we use the Nesterov’s optimal gradient method to optimize the kernel weights. One remarkable advantage of solving in the primal is that it achieves much faster convergence rate than solving in the dual and does not require a two-stage algorithm even for the single kernel LapSVM. Introducing the Laplacian regularizer, we also extend our primal method to semi-supervised scenario. Extensive experiments on some UCI benchmarks have shown that the proposed algorithm converges rapidly and achieves competitive accuracy.     

Multiple-kernel-learning-based extreme learning machine for classification design

The extreme learning machine (ELM) is a new method for using single hidden layer feed-forward networks with a much simpler training method. While conventional kernel-based classifiers are based on a single kernel, in reality, it is often desirable to base classifiers on combinations of multiple kernels. In this paper, we propose the issue of multiple-kernel learning (MKL) for ELM by formulating it as a semi-infinite linear programming. We further extend this idea by integrating with techniques of MKL. The kernel function in this ELM formulation no longer needs to be fixed, but can be automatically learned as a combination of multiple kernels. Two formulations of multiple-kernel classifiers are proposed. The first one is based on a convex combination of the given base kernels, while the second one uses a convex combination of the so-called equivalent kernels. Empirically, the second formulation is particularly competitive. Experiments on a large number of both toy and real-world data sets (including high-magnification sampling rate image data set) show that the resultant classifier is fast and accurate and can also be easily trained by simply changing linear program.     

LSSVR ensemble learning with uncertain parameters for crude oil price forecasting

Least squares support vector regression (LSSVR) is an effective and competitive approach for crude oil price prediction, but its performance suffers from parameter sensitivity and long tuning time. This paper considers the user-defined parameters as uncertain (or random) factors to construct an LSSVR ensemble learning paradigm, by taking four major steps. First, probability distributions of the user-defined parameters in LSSVR are designed using grid method for low upper bound estimation (LUBE). Second, random sets of parameters are generated according to the designed probability distributions to formulate diverse individual LSSVR members. Third, each individual member is applied to individual prediction. Finally, all individual results are combined to the final output via ensemble weighted averaging, with probabilities measuring the corresponding weights. The computational experiment using the crude oil spot price of West Texas Intermediate (WTI) verifies the effectiveness of the proposed LSSVR ensemble learning paradigm with uncertain parameters compared with some existing LSSVR variants (using other popular parameters selection algorithms), in terms of prediction accuracy and time-saving.     

基于LSSVR和LSTM的多模型优化集成负荷预测

针对负荷需求受多源因素影响和现有单模型预测方法精度较低的问题,提出了一种基于最小二乘支持向量回归(LSSVR)和长短期记忆循环神经网络(LSTM)的多模型优化集成负荷预测方法。首先探究负荷相关特征的特性并由互信息进行特征选择,获取最优特征集。在此基础上采用随机抽样(bootstrap)生成多个训练集,然后使用具有良好预测能力的LSSVR和LSTM模型对多个训练集分别进行预测。利用混沌粒子群优化算法(CPSO)进一步提高模型预测精度。最后,在决策阶段中使用偏最小二乘回归(PLSR)组合各个子模型的最优预测输出并提供最终预测结果。对真实电网数据进行了仿真,并与其它预测方法进行了比较。本文所提方法的应用范围广泛且预测精度提高显著。     

Improving project-profit prediction using a two-stage forecasting system

Accurate project-profit prediction is a crucial issue because it can provide an early feasibility estimate for the project. In order to achieve accurate project-profit prediction, this study developed a novel two-stage forecasting system. In stage one, the proposed forecasting system adopts fuzzy clustering technology, fuzzy c-means (FCM) and kernel fuzzy c-means (KFCM), for the correct grouping of different projects. In stage two, least-squares support vector regression (LSSVR) technology is employed for forecasting the project-profit in different project groups, respectively. Moreover, genetic algorithms (GA) were simultaneously used to select the parameters of the LSSVR. The project data come from a real enterprise in Taiwan. In this study, some forecasting methodologies are also compared, for instance Generalized Regression Neural Network (GRNN), Radial Basis Function Neural Networks (RBFNN), and Back Propagation Neural Network (BPNN), to predict project-profit in this real case. Empirical results indicate that the two-stage forecasting system (FCM+LSSVR and KFCM+LSSVR) has superior performance in terms of forecasting accuracy, compared to other methods. Furthermore, in observing the results of the two-stage forecasting system, it can be seen that FCM+LSSVR can achieve superior performance, and KFCM+LSSVR can achieve consistently good performance. Therefore, based on the empirical results, the two-stage forecasting system was verified to efficiently provide credible predictions for project-profit forecasting.     

Multiple empirical kernel learning based on local information

The traditional multiple kernel learning (MKL) is usually based on implicit kernel mapping and adopts a certain combination of kernels instead of a single kernel. MKL has been demonstrated to have a significant advantage to the single-kernel learning. Although MKL sets different weights to different kernels, the weights are not changed over the whole input space. This weight setting might not been fit for those data with some underlying local distributions. In order to solve this problem, Gönen and Alpayd?n (2008) introduced a localizing gating model into the traditional MKL framework so as to assign different weights to a kernel in different regions of the input space. In this paper, we also integrate the localizing gating model into our previous work named MultiK-MHKS that is an effective multiple empirical kernel learning. In doing so, we can get multiple localized empirical kernel learning named MLEKL. Our contribution is that we first establish a localized formulation in the empirical kernel learning framework. The experimental results on benchmark data sets validate the effectiveness of the proposed MLEKL.     

Parallel multiple kernel learning: a hybrid alternating direction method of multipliers

Multiple kernel learning (MKL) has recently become a hot topic in kernel methods. However, many MKL algorithms suffer from high computational cost. Moreover, standard MKL algorithms face the challenge of the rapid development of distributed computational environment such as cloud computing. In this study, a framework for parallel multiple kernel learning (PMKL) using hybrid alternating direction method of multipliers (H-ADMM) is developed to integrate the MKL algorithms and the multiprocessor system. The global problem with multiple kernel is divided into multiple local problems each of which is optimized in a local processor with a single kernel. An H-ADMM is proposed to make the local processors coordinate with each other to achieve the global optimal solution. The results of computational experiments show that PMKL exhibits high classification accuracy and fast computational speed.     

Deep multilayer multiple kernel learning

Multiple kernel learning (MKL) approach has been proposed for kernel methods and has shown high performance for solving some real-world applications. It consists on learning the optimal kernel from one layer of multiple predefined kernels. Unfortunately, this approach is not rich enough to solve relatively complex problems. With the emergence and the success of the deep learning concept, multilayer of multiple kernel learning (MLMKL) methods were inspired by the idea of deep architecture. They are introduced in order to improve the conventional MKL methods. Such architectures tend to learn deep kernel machines by exploring the combinations of multiple kernels in a multilayer structure. However, existing MLMKL methods often have trouble with the optimization of the network for two or more layers. Additionally, they do not always outperform the simplest method of combining multiple kernels (i.e., MKL). In order to improve the effectiveness of MKL approaches, we introduce, in this paper, a novel backpropagation MLMKL framework. Specifically, we propose to optimize the network over an adaptive backpropagation algorithm. We use the gradient ascent method instead of dual objective function, or the estimation of the leave-one-out error. We test our proposed method through a large set of experiments on a variety of benchmark data sets. We have successfully optimized the system over many layers. Empirical results over an extensive set of experiments show that our algorithm achieves high performance compared to the traditional MKL approach and existing MLMKL methods.     

基于振动频谱的磨机负荷在线软测量建模

针对磨机负荷(ML)软测量模型难以适应磨矿过程的时变特性,模型需要依据工况实时在线更新的问题,基于磨机简体振动频谱,通过递归主元分析(RPCA)和在线最小二乘支持向量回归机(LSSVR)的集成,提出了ML参数(料球比、矿浆浓度、充填率)在线软测量方法.首先,针对训练样本,采用主元分析(PCA)分别提取振动频谱在低、中、高频段的谱主元;然后以串行组合后的谱主元为输入,采用LSSVR方法构造ML参数离线软测量模型;最后,采用旧模型完成预测后,应用RPCA及在线LSSVR算法分别递归更新模型的输入和模型的回归参数,从而实现了ML软测量模型的在线更新.实验结果表明,该软测量方法与其它常规方法相比具有较高的精度和更好的预测性能.     

A hybrid grid-GA-based LSSVR learning paradigm for crude oil price forecasting

In order to effectively model crude oil spot price with inherently high complexity, a hybrid learning paradigm integrating least squares support vector regression (LSSVR) with a hybrid optimization searching approach for the parameters selection in the LSSVR [consisting of grid method and genetic algorithm (GA)], i.e., a hybrid grid-GA-based LSSVR model, is proposed in this study. In the proposed hybrid learning paradigm, the grid method, a simple but efficient searching method, is first applied to roughly but rapidly determine the proper boundaries of the parameters in the LSSVR; then, the GA, an effective and powerful intelligent searching algorithm, is further implemented to select the most suitable parameters. For illustration and verification, the proposed learning paradigm is used to predict the crude oil spot prices of the West Texas Intermediate and the Brent markets. The empirical results demonstrate that the proposed hybrid grid-GA-based LSSVR learning paradigm can outperform its benchmarking models (including some popular forecasting techniques and similar LSSVRs with other parameter searching algorithms) in terms of both prediction accuracy and time-savings, indicating that it can be utilized as one effective forecasting tool for crude oil price with high volatility and irregularity.