M<Emphasis Type="SmallCaps">l-rbf</Emphasis>: RBF Neural Networks for Multi-Label Learning

Multi-label learning deals with the problem where each instance is associated with multiple labels simultaneously. The task of this learning paradigm is to predict the label set for each unseen instance, through analyzing training instances with known label sets. In this paper, a neural network based multi-label learning algorithm named Ml-rbf is proposed, which is derived from the traditional radial basis function (RBF) methods. Briefly, the first layer of an Ml-rbf neural network is formed by conducting clustering analysis on instances of each possible class, where the centroid of each clustered groups is regarded as the prototype vector of a basis function. After that, second layer weights of the Ml-rbf neural network are learned by minimizing a sum-of-squares error function. Specifically, information encoded in the prototype vectors corresponding to all classes are fully exploited to optimize the weights corresponding to each specific class. Experiments on three real-world multi-label data sets show that Ml-rbf achieves highly competitive performance to other well-established multi-label learning algorithms.     

A method of learning weighted similarity function to improve the performance of nearest neighbor

The performance of Nearest Neighbor (NN) classifier is known to be sensitive to the distance (or similarity) function used in classifying a test instance. Another major disadvantage of NN is that it uses all training instances in the generalization phase. This can cause slow execution speed and high storage requirement when dealing with large datasets. In the past research, many solutions have been proposed to handle one or both of the above problems. In the scheme proposed in this paper, we tackle both of these problems by assigning a weight to each training instance. The weight of a training instance is used in the generalization phase to calculate the distance (or similarity) of a query pattern to that instance. The basic NN classifier can be viewed as a special case of this scheme that treats all instances equally (by assigning equal weight to all training instances). Using this form of weighted similarity measure, we propose a learning algorithm that attempts to maximize the leave-one-out (LV1) classification rate of the NN rule by adjusting the weights of the training instances. At the same time, the algorithm reduces the size of the training set and can be viewed as a powerful instance reduction technique. An instance having zero weight is not used in the generalization phase and can be virtually removed from the training set. We show that our scheme has comparable or better performance than some recent methods proposed in the literature for the task of learning the distance function and/or prototype reduction.     

Fast and improved backpropagation learning of multi-layer artificial neural network using adaptive activation function

In the conventional backpropagation (BP) learning algorithm used for the training of the connecting weights of the artificial neural network (ANN), a fixed slope−based sigmoidal activation function is used. This limitation leads to slower training of the network because only the weights of different layers are adjusted using the conventional BP algorithm. To accelerate the rate of convergence during the training phase of the ANN, in addition to updates of weights, the slope of the sigmoid function associated with artificial neuron can also be adjusted by using a newly developed learning rule. To achieve this objective, in this paper, new BP learning rules for slope adjustment of the activation function associated with the neurons have been derived. The combined rules both for connecting weights and slopes of sigmoid functions are then applied to the ANN structure to achieve faster training. In addition, two benchmark problems: classification and nonlinear system identification are solved using the trained ANN. The results of simulation-based experiments demonstrate that, in general, the proposed new BP learning rules for slope and weight adjustments of ANN provide superior convergence performance during the training phase as well as improved performance in terms of root mean square error and mean absolute deviation for classification and nonlinear system identification problems.     

Simultaneous optimization of artificial neural networks for financial forecasting

Artificial neural networks (ANNs) have been popularly applied for stock market prediction, since they offer superlative learning ability. However, they often result in inconsistent and unpredictable performance in the prediction of noisy financial data due to the problems of determining factors involved in design. Prior studies have suggested genetic algorithm (GA) to mitigate the problems, but most of them are designed to optimize only one or two architectural factors of ANN. With this background, the paper presents a global optimization approach of ANN to predict the stock price index. In this study, GA optimizes multiple architectural factors and feature transformations of ANN to relieve the limitations of the conventional backpropagation algorithm synergistically. Experiments show our proposed model outperforms conventional approaches in the prediction of the stock price index.     

ML-KNN: A lazy learning approach to multi-label learning

Multi-label learning originated from the investigation of text categorization problem, where each document may belong to several predefined topics simultaneously. In multi-label learning, the training set is composed of instances each associated with a set of labels, and the task is to predict the label sets of unseen instances through analyzing training instances with known label sets. In this paper, a multi-label lazy learning approach named ML-KNN is presented, which is derived from the traditional K-nearest neighbor (KNN) algorithm. In detail, for each unseen instance, its K nearest neighbors in the training set are firstly identified. After that, based on statistical information gained from the label sets of these neighboring instances, i.e. the number of neighboring instances belonging to each possible class, maximum a posteriori (MAP) principle is utilized to determine the label set for the unseen instance. Experiments on three different real-world multi-label learning problems, i.e. Yeast gene functional analysis, natural scene classification and automatic web page categorization, show that ML-KNN achieves superior performance to some well-established multi-label learning algorithms.     

Bayesian multi-instance multi-label learning using Gaussian process prior

Multi-instance multi-label learning (MIML) is a newly proposed framework, in which the multi-label problems are investigated by representing each sample with multiple feature vectors named instances. In this framework, the multi-label learning task becomes to learn a many-to-many relationship, and it also offers a possibility for explaining why a concerned sample has the certain class labels. The connections between instances and labels as well as the correlations among labels are equally crucial information for MIML. However, the existing MIML algorithms can rarely exploit them simultaneously. In this paper, a new MIML algorithm is proposed based on Gaussian process. The basic idea is to suppose a latent function with Gaussian process prior in the instance space for each label and infer the predictive probability of labels by integrating over uncertainties in these functions using the Bayesian approach, so that the connection between instances and every label can be exploited by defining a likelihood function and the correlations among labels can be identified by the covariance matrix of the latent functions. Moreover, since different relationships between instances and labels can be captured by defining different likelihood functions, the algorithm may be used to deal with the problems with various multi-instance assumptions. Experimental results on several benchmark data sets show that the proposed algorithm is valid and can achieve superior performance to the existing ones.     

A filter model for feature subset selection based on genetic algorithm

This paper describes a novel feature subset selection algorithm, which utilizes a genetic algorithm (GA) to optimize the output nodes of trained artificial neural network (ANN). The new algorithm does not depend on the ANN training algorithms or modify the training results. The two groups of weights between input-hidden and hidden-output layers are extracted after training the ANN on a given database. The general formula for each output node (class) of ANN is then generated. This formula depends only on input features because the two groups of weights are constant. This dependency is represented by a non-linear exponential function. The GA is involved to find the optimal relevant features, which maximize the output function for each class. The dominant features in all classes are the features subset to be selected from the input feature group.     

Instance selection method for improving graph-based semi-supervised learning

Graph-based semi-supervised learning is an important semi-supervised learning paradigm. Although graph-based semi-supervised learning methods have been shown to be helpful in various situations, they may adversely affect performance when using unlabeled data. In this paper, we propose a new graph-based semi-supervised learning method based on instance selection in order to reduce the chances of performance degeneration. Our basic idea is that given a set of unlabeled instances, it is not the best approach to exploit all the unlabeled instances; instead, we should exploit the unlabeled instances that are highly likely to help improve the performance, while not taking into account the ones with high risk. We develop both transductive and inductive variants of our method. Experiments on a broad range of data sets show that the chances of performance degeneration of our proposed method are much smaller than those of many state-of-the-art graph-based semi-supervised learning methods.     

Evolving neural network using real coded genetic algorithm for permeability estimation of the reservoir

In this work we investigate how artificial neural network (ANN) evolution with genetic algorithm (GA) improves the reliability and predictability of artificial neural network. This strategy is applied to predict permeability of Mansuri Bangestan reservoir located in Ahwaz, Iran utilizing available geophysical well log data. Our methodology utilizes a hybrid genetic algorithm–neural network strategy (GA–ANN). The proposed algorithm combines the local searching ability of the gradient–based back-propagation (BP) strategy with the global searching ability of genetic algorithms. Genetic algorithms are used to decide the initial weights of the gradient decent methods so that all the initial weights can be searched intelligently. The genetic operators and parameters are carefully designed and set avoiding premature convergence and permutation problems. For an evaluation purpose, the performance and generalization capabilities of GA–ANN are compared with those of models developed with the common technique of BP. The results demonstrate that carefully designed genetic algorithm-based neural network outperforms the gradient descent-based neural network.     

Active learning through density clustering

Active learning is used for classification when labeling data are costly, while the main challenge is to identify the critical instances that should be labeled. Clustering-based approaches take advantage of the structure of the data to select representative instances. In this paper, we developed the active learning through density peak clustering (ALEC) algorithm with three new features. First, a master tree was built to express the relationships among the nodes and assist the growth of the cluster tree. Second, a deterministic instance selection strategy was designed using a new importance measure. Third, tri-partitioning was employed to determine the action to be taken on each instance during iterative clustering, labeling, and classifying. Experiments were performed with 14 datasets to compare against state-of-the-art active learning algorithms. Results demonstrated that the new algorithm had higher classification accuracy using the same number of labeled data.     

Parallelizing Feature Selection

Classification is a key problem in machine learning/data mining. Algorithms for classification have the ability to predict the class of a new instance after having been trained on data representing past experience in classifying instances. However, the presence of a large number of features in training data can hurt the classification capacity of a machine learning algorithm. The Feature Selection problem involves discovering a subset of features such that a classifier built only with this subset would attain predictive accuracy no worse than a classifier built from the entire set of features. Several algorithms have been proposed to solve this problem. In this paper we discuss how parallelism can be used to improve the performance of feature selection algorithms. In particular, we present, discuss and evaluate a coarse-grained parallel version of the feature selection algorithm FortalFS. This algorithm performs well compared with other solutions and it has certain characteristics that makes it a good candidate for parallelization. Our parallel design is based on the master--slave design pattern. Promising results show that this approach is able to achieve near optimum speedups in the context of Amdahl's Law.     

A Cognitive Bias Approach to Feature Selection and Weighting for Case-Based Learners

Research in psychology, psycholinguistics, and cognitive science has discovered and examined numerous psychological constraints on human information processing. Short term memory limitations, a focus of attention bias, and a preference for the use of temporally recent information are three examples. This paper shows that psychological constraints such as these can be used effectively as domain-independent sources of bias to guide feature set selection and weighting for case-based learning algorithms.We first show that cognitive biases can be automatically and explicitly encoded into the baseline instance representation: each bias modifies the representation by changing features, deleting features, or modifying feature weights. Next, we investigate the related problems of cognitive bias selection and cognitive bias interaction for the feature weighting approach. In particular, we compare two cross-validation algorithms for bias selection that make different assumptions about the independence of individual component biases. In evaluations on four natural language learning tasks, we show that the bias selection algorithms can determine which cognitive bias or biases are relevant for each learning task and that the accuracy of the case-based learning algorithm improves significantly when the selected bias(es) are incorporated into the baseline instance representation.     

Learning-based multi-pass adaptive scheduling for a dynamic manufacturing cell environment

Because the essential attributes are uncertain in a dynamic manufacturing cell environment, to select a near-optimal subset of manufacturing attributes to enhance the generalization ability of knowledge bases remains a critical, unresolved issue for classical artificial neural network-based (ANN-based) multi-pass adaptive scheduling (MPAS). To resolve this problem, this study develops a hybrid genetic /artificial neural network (GA/ANN) approach for ANN-based MPAS systems. The hybrid GA/ANN approach is used to evolve an optimal subset of system attributes from a large set of candidate manufacturing system attributes and, simultaneously, to determine configuration and learning parameters of the ANN according to various performance measures. In the GA/ANN-based MPAS approach, for a given feature subset and the corresponding topology and learning parameters of an ANN decoded by a GA, an ANN was applied to evaluate the fitness in the GA process and to generate the MPAS knowledge base used for adaptive scheduling control mechanisms. The results demonstrate that the proposed GA/ANN-based MPAS approach has, according to various performance criteria, a better system performance over a long period of time than those obtained with classical machine learning-based MPAS approaches and the heuristic individual dispatching rules.     

An efficient hybrid learning algorithm for neural network–based speech recognition systems on FPGA chip

This paper implemented an artificial neural network (ANN) on a field programmable gate array (FPGA) chip for Mandarin speech measurement and recognition of nonspecific speaker. A three-layer hybrid learning algorithm (HLA), which combines genetic algorithm (GA) and steepest descent method, was proposed to fulfill a faster global search of optimal weights in ANN. Some other popular evolutionary algorithms, such as differential evolution, particle swarm optimization and improve GA, were compared to the proposed HLA. It can be seen that the proposed HLA algorithm outperforms the other algorithms. Finally, the designed system was implemented on an FPGA chip with an SOC architecture to measure and recognize the speech signals.     

Instances selection algorithm by ensemble margin

Abstract

The main limit of data mining algorithms is their inability to deal with the huge amount of available data in a reasonable processing time. A solution of producing fast and accurate results is instances and features selection. This process eliminates noisy or redundant data in order to reduce the storage and computational cost without performances degradation. In this paper, a new instance selection approach called Ensemble Margin Instance Selection (EMIS) algorithm is proposed. This approach is based on the ensemble margin. To evaluate our approach, we have conducted several experiments on different real-world classification problems from UCI Machine learning repository. The pixel-based image segmentation is a field where the storage requirement and computational cost of applied model become higher. To solve these limitations we conduct a study based on the application of EMIS and other instance selection techniques for the segmentation and automatic recognition of white blood cells WBC (nucleus and cytoplasm) in cytological images.     

IBMvSVM: An instance-based multi-view SVM algorithm for classification

As an emerging research direction of machine learning, the multi-view learning (MVL) pays attention to the tasks that learn from datasets with several distinct views to achieve better generalization performance. Recently, various Support Vector Machine (SVM)-based algorithms with solid theoretical foundation have been proposed for MVL. However, there is a constraining assumption for these algorithms, i.e., in the learning process, different views are important equally for an instance in a data set, and the same view is important equally for all instances in a data set. In fact, an instance generally has different adaptability to different views, namely, the degree to which the information from different views accurately describes the instance varies. And naturally, different instances in a data set also have different adaptability to the same view. In this paper, the concept of view vector of each instance is proposed first, which quantitatively describes the adaptability of a specific instance to different views. It also reflects the characteristics of different instances that some instances are more suitable to be represented by a view, while others tend to be better represented by another view. Then, a new instance-based multi-view SVM algorithm, named IBMvSVM, is proposed by building the view vector of each instance into the multi-view SVM learning. IBMvSVM focuses on characteristics of each instance itself in different views rather than treating them equally. Experiments performed on 48 multi-view datasets reveal the superiority of IBMvSVM algorithm on generalization against several recently state-of-the-art MVL algorithms.

     

A divide-and-conquer recursive approach for scaling up instance selection algorithms

Instance selection is becoming more and more relevant due to the huge amount of data that is being constantly produced. However, although current algorithms are useful for fairly large datasets, scaling problems are found when the number of instances is of hundreds of thousands or millions. In the best case, these algorithms are of efficiency O(n ²), n being the number of instances. When we face huge problems, scalability is an issue, and most algorithms are not applicable. This paper presents a divide-and-conquer recursive approach to the problem of instance selection for instance based learning for very large problems. Our method divides the original training set into small subsets where the instance selection algorithm is applied. Then the selected instances are rejoined in a new training set and the same procedure, partitioning and application of an instance selection algorithm, is repeated. In this way, our approach is based on the philosophy of divide-and-conquer applied in a recursive manner. The proposed method is able to match, and even improve, for the case of storage reduction, the results of well-known standard algorithms with a very significant reduction of execution time. An extensive comparison in 30 datasets form the UCI Machine Learning Repository shows the usefulness of our method. Additionally, the method is applied to 5 huge datasets with from 300,000 to more than a million instances, with very good results and fast execution time.     

Towards objective measures of algorithm performance across instance space

This paper tackles the difficult but important task of objective algorithm performance assessment for optimization. Rather than reporting average performance of algorithms across a set of chosen instances, which may bias conclusions, we propose a methodology to enable the strengths and weaknesses of different optimization algorithms to be compared across a broader instance space. The results reported in a recent Computers and Operations Research paper comparing the performance of graph coloring heuristics are revisited with this new methodology to demonstrate (i) how pockets of the instance space can be found where algorithm performance varies significantly from the average performance of an algorithm; (ii) how the properties of the instances can be used to predict algorithm performance on previously unseen instances with high accuracy; and (iii) how the relative strengths and weaknesses of each algorithm can be visualized and measured objectively.     

Internal representation in neural networks used for classification of patient anaesthetic states and dosage.

In this study we aimed to explore the ability of artificial neural networks (ANN) to classify patient anaesthetic states and dosage. Surgical data obtained under different states of anaesthesia and dose levels were modelled via this approach. It is shown that inferential parameters can be used to determine the patient anaesthetic states and drug dosage. In addition to demonstrating the capability of ANN for classification we were interested in the internal representations that are developed automatically by networks while they are learning their processing task. An unsupervised learning procedure of clustering via which the classes are inferred from the data and a supervised learning technique of discrimination via which to construct a classification of the known categories were applied to analyse the performance of the ANN. Discriminant analysis (DA) was also utilised to optimise the network architecture.     

Non-linear variable selection for artificial neural networks using partial mutual information

Artificial neural networks (ANNs) have been widely used to model environmental processes. The ability of ANN models to accurately represent the complex, non-linear behaviour of relatively poorly understood processes makes them highly suited to this task. However, the selection of an appropriate set of input variables during ANN development is important for obtaining high-quality models. This can be a difficult task when considering that many input variable selection (IVS) techniques fail to perform adequately due to an underlying assumption of linearity, or due to redundancy within the available data.This paper focuses on a recently proposed IVS algorithm, based on estimation of partial mutual information (PMI), which can overcome both of these issues and is considered highly suited to the development of ANN models. In particular, this paper addresses the computational efficiency and accuracy of the algorithm via the formulation and evaluation of alternative techniques for determining the significance of PMI values estimated during selection. Furthermore, this paper presents a rigorous assessment of the PMI-based algorithm and clearly demonstrates the superior performance of this non-linear IVS technique in comparison to linear correlation-based techniques.