Performance evaluation of evolutionary multiobjective optimization algorithms for multiobjective fuzzy genetics-based machine learning

Recently, evolutionary multiobjective optimization (EMO) algorithms have been utilized for the design of accurate and interpretable fuzzy rule-based systems. This research area is often referred to as multiobjective genetic fuzzy systems (MoGFS), where EMO algorithms are used to search for non-dominated fuzzy rule-based systems with respect to their accuracy and interpretability. In this paper, we examine the ability of EMO algorithms to efficiently search for Pareto optimal or near Pareto optimal fuzzy rule-based systems for classification problems. We use NSGA-II (elitist non-dominated sorting genetic algorithm), its variants, and MOEA/D (multiobjective evolutionary algorithm based on decomposition) in our multiobjective fuzzy genetics-based machine learning (MoFGBML) algorithm. Classification performance of obtained fuzzy rule-based systems by each EMO algorithm is evaluated for training data and test data under various settings of the available computation load and the granularity of fuzzy partitions. Experimental results in this paper suggest that reported classification performance of MoGFS in the literature can be further improved using more computation load, more efficient EMO algorithms, and/or more antecedent fuzzy sets from finer fuzzy partitions.     

Balance between genetic search and local search in memetic algorithms for multiobjective permutation flowshop scheduling

This paper shows how the performance of evolutionary multiobjective optimization (EMO) algorithms can be improved by hybridization with local search. The main positive effect of the hybridization is the improvement in the convergence speed to the Pareto front. On the other hand, the main negative effect is the increase in the computation time per generation. Thus, the number of generations is decreased when the available computation time is limited. As a result, the global search ability of EMO algorithms is not fully utilized. These positive and negative effects are examined by computational experiments on multiobjective permutation flowshop scheduling problems. Results of our computational experiments clearly show the importance of striking a balance between genetic search and local search. In this paper, we first modify our former multiobjective genetic local search (MOGLS) algorithm by choosing only good individuals as initial solutions for local search and assigning an appropriate local search direction to each initial solution. Next, we demonstrate the importance of striking a balance between genetic search and local search through computational experiments. Then we compare the modified MOGLS with recently developed EMO algorithms: the strength Pareto evolutionary algorithm and revised nondominated sorting genetic algorithm. Finally, we demonstrate that a local search can be easily combined with those EMO algorithms for designing multiobjective memetic algorithms.     

Multiobjectivity and Complexity in Embodied Cognition

We propose a novel perspective on the use of evolutionary multiobjective optimization (EMO) as a paradigm for evolving embodied organisms and as a framework for characterizing complexity. The paper demonstrates novel experiments that show the power of EMO in generating robots with different morphologies, yet with very similar locomotion abilities. The proposed framework for comparing the complexity of an object across different complexity measures allowed meaningful and quantifiable comparisons between the evolved organisms. We show empirically that the partial order feature inherited in the Pareto concept exhibits characteristics which are suitable for comparing between the complexities of artificially evolved embodied organisms.     

Cooperative coevolution of artificial neural network ensembles for pattern classification

This paper presents a cooperative coevolutive approach for designing neural network ensembles. Cooperative coevolution is a recent paradigm in evolutionary computation that allows the effective modeling of cooperative environments. Although theoretically, a single neural network with a sufficient number of neurons in the hidden layer would suffice to solve any problem, in practice many real-world problems are too hard to construct the appropriate network that solve them. In such problems, neural network ensembles are a successful alternative. Nevertheless, the design of neural network ensembles is a complex task. In this paper, we propose a general framework for designing neural network ensembles by means of cooperative coevolution. The proposed model has two main objectives: first, the improvement of the combination of the trained individual networks; second, the cooperative evolution of such networks, encouraging collaboration among them, instead of a separate training of each network. In order to favor the cooperation of the networks, each network is evaluated throughout the evolutionary process using a multiobjective method. For each network, different objectives are defined, considering not only its performance in the given problem, but also its cooperation with the rest of the networks. In addition, a population of ensembles is evolved, improving the combination of networks and obtaining subsets of networks to form ensembles that perform better than the combination of all the evolved networks. The proposed model is applied to ten real-world classification problems of a very different nature from the UCI machine learning repository and proben1 benchmark set. In all of them the performance of the model is better than the performance of standard ensembles in terms of generalization error. Moreover, the size of the obtained ensembles is also smaller.     

Selection-fusion approach for classification of datasets with missing values

This paper proposes a new approach based on missing value pattern discovery for classifying incomplete data. This approach is particularly designed for classification of datasets with a small number of samples and a high percentage of missing values where available missing value treatment approaches do not usually work well. Based on the pattern of the missing values, the proposed approach finds subsets of samples for which most of the features are available and trains a classifier for each subset. Then, it combines the outputs of the classifiers. Subset selection is translated into a clustering problem, allowing derivation of a mathematical framework for it. A trade off is established between the computational complexity (number of subsets) and the accuracy of the overall classifier. To deal with this trade off, a numerical criterion is proposed for the prediction of the overall performance. The proposed method is applied to seven datasets from the popular University of California, Irvine data mining archive and an epilepsy dataset from Henry Ford Hospital, Detroit, Michigan (total of eight datasets). Experimental results show that classification accuracy of the proposed method is superior to those of the widely used multiple imputations method and four other methods. They also show that the level of superiority depends on the pattern and percentage of missing values.     

Sensitivity and specificity based multiobjective approach for feature selection: Application to cancer diagnosis

The study of the sensitivity and the specificity of a classification test constitute a powerful kind of analysis since it provides specialists with very detailed information useful for cancer diagnosis. In this work, we propose the use of a multiobjective genetic algorithm for gene selection of Microarray datasets. This algorithm performs gene selection from the point of view of the sensitivity and the specificity, both used as quality indicators of the classification test applied to the previously selected genes. In this algorithm, the classification task is accomplished by Support Vector Machines; in addition a 10-Fold Cross-Validation is applied to the resulting subsets. The emerging behavior of all these techniques used together is noticeable, since this approach is able to offer, in an original and easy way, a wide range of accurate solutions to professionals in this area. The effectiveness of this approach is proved on public cancer datasets by working out new and promising results. A comparative analysis of our approach using two and three objectives, and with other existing algorithms, suggest that our proposal is highly appropriate for solving this problem.     

On Set-Based Multiobjective Optimization

Assuming that evolutionary multiobjective optimization (EMO) mainly deals with set problems, one can identify three core questions in this area of research: 1) how to formalize what type of Pareto set approximation is sought; 2) how to use this information within an algorithm to efficiently search for a good Pareto set approximation; and 3) how to compare the Pareto set approximations generated by different optimizers with respect to the formalized optimization goal. There is a vast amount of studies addressing these issues from different angles, but so far only a few studies can be found that consider all questions under one roof. This paper is an attempt to summarize recent developments in the EMO field within a unifying theory of set-based multiobjective search. It discusses how preference relations on sets can be formally defined, gives examples for selected user preferences, and proposes a general preference-independent hill climber for multiobjective optimization with theoretical convergence properties. Furthermore, it shows how to use set preference relations for statistical performance assessment and provides corresponding experimental results. The proposed methodology brings together preference articulation, algorithm design, and performance assessment under one framework and thereby opens up a new perspective on EMO.      

Bandit-based cooperative coevolution for tackling contribution imbalance in large-scale optimization problems

This paper addresses the issue of computational resource allocation within the context of cooperative coevolution. Cooperative coevolution typically works by breaking a problem down into smaller subproblems (or components) and coevolving them in a round-robin fashion, resulting in a uniform resource allocation among its components. Despite its success on a wide range of problems, cooperative coevolution struggles to perform efficiently when its components do not contribute equally to the overall objective value. This is of crucial importance on large-scale optimization problems where such difference are further magnified. To resolve this imbalance problem, we extend the standard cooperative coevolution to a new generic framework capable of learning the contribution of each component using multi-armed bandit techniques. The new framework allocates the computational resources to each component proportional to their contributions towards improving the overall objective value. This approach results in a more economical use of the limited computational resources. We study different aspects of the proposed framework in the light of extensive experiments. Our empirical results confirm that even a simple bandit-based credit assignment scheme can significantly improve the performance of cooperative coevolution on large-scale continuous problems, leading to competitive performance as compared to the state-of-the-art algorithms.     

HColonies: a new hybrid metaheuristic for medical data classification

Medical data feature a number of characteristics that make their classification a complex task. Yet, the societal significance of the subject and the computational challenge it presents has caused the classification of medical datasets to be a popular research area. A new hybrid metaheuristic is presented for the classification task of medical datasets. The hybrid ant–bee colonies (HColonies) consists of two phases: an ant colony optimization (ACO) phase and an artificial bee colony (ABC) phase. The food sources of ABC are initialized into decision lists, constructed during the ACO phase using different subsets of the training data. The task of the ABC is to optimize the obtained decision lists. New variants of the ABC operators are proposed to suit the classification task. Results on a number of benchmark, real-world medical datasets show the usefulness of the proposed approach. Classification models obtained feature good predictive accuracy and relatively small model size.     

Automated root cause identification of security alerts: Evaluation in a SaaS Cloud

The analysis of the security alerts collected during the system operations is a crucial task to initiate effective responses against attacks and intentional system misuse. A variety of monitors are today available to generate security alerts, such as intrusion detection systems, network audit, vulnerability scans, and event logs. While the amount of alerts generated by the security monitors represents a goldmine of information, the ever-increasing volume and heterogeneity of the collected alerts pose a major threat to timely security analysis and forensic activities conducted by the operations team.This paper proposes a framework consisting of a filter and a decision tree to address large volumes of security alerts and to support the automated identification of the root causes of the alerts. The framework adopts both term weighting and conceptual clustering approaches to fill the gap between the unstructured textual alerts and the formalization of the decision tree. We evaluated the framework by analyzing two security datasets in a production SaaS Cloud, which generates an average volume of 800 alerts/day. The framework significantly reduced the volume of alerts and inferred the root causes of around 98.8% of alerts with no human intervention with respect to the datasets available in this study. More important, we leveraged the output of the framework to provide a classification of the root causes of the alerts in the target SaaS Cloud.     

CurveNet: Curvature-Based Multitask Learning Deep Networks for 3D Object Recognition

In computer vision fields, 3D object recognition is one of the most important tasks for many real-world applications. Three-dimensional convolutional neural networks (CNNs) have demonstrated their advantages in 3D object recognition. In this paper, we propose to use the principal curvature directions of 3D objects (using a CAD model) to represent the geometric features as inputs for the 3D CNN. Our framework, namely CurveNet, learns perceptually relevant salient features and predicts object class labels. Curvature directions incorporate complex surface information of a 3D object, which helps our framework to produce more precise and discriminative features for object recognition. Multitask learning is inspired by sharing features between two related tasks, where we consider pose classification as an auxiliary task to enable our CurveNet to better generalize object label classification. Experimental results show that our proposed framework using curvature vectors performs better than voxels as an input for 3D object classification. We further improved the performance of CurveNet by combining two networks with both curvature direction and voxels of a 3D object as the inputs. A Cross-Stitch module was adopted to learn effective shared features across multiple representations. We evaluated our methods using three publicly available datasets and achieved competitive performance in the 3D object recognition task.      

Effects of nongeometric binary crossover on multiobjective 0/1 knapsack problems

Standard binary crossover operators (e.g., one-point, two-point, and uniform) tend to decrease the diversity of solutions while they improve the convergence to the Pareto front. This is because standard binary crossover operators, which are called geometric crossovers, always generate an offspring in the line segment between its parents under the Hamming distance in the genotype space. In our former study, we have already proposed a nongeometric binary crossover operator to generate an offspring outside the line segment between its parents. In this article, we examine the effect of our crossover operator on the performance of evolutionary multiobjective optimization (EMO) algorithms through computational experiments on various multiobjective knapsack problems. Experimental results show that our crossover operator improves the search ability of EMO algorithms for a wide range of test problems. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

Symbiotic coevolutionary genetic programming: a benchmarking study under large attribute spaces

Classification under large attribute spaces represents a dual learning problem in which attribute subspaces need to be identified at the same time as the classifier design is established. Embedded as opposed to filter or wrapper methodologies address both tasks simultaneously. The motivation for this work stems from the observation that team based approaches to Genetic Programming (GP) have the potential to design multiple classifiers per class—each with a potentially unique attribute subspace—without recourse to filter or wrapper style preprocessing steps. Specifically, competitive coevolution provides the basis for scaling the algorithm to data sets with large instance counts; whereas cooperative coevolution provides a framework for problem decomposition under a bid-based model for establishing program context. Symbiosis is used to separate the tasks of team/ensemble composition from the design of specific team members. Team composition is specified in terms of a combinatorial search performed by a Genetic Algorithm (GA); whereas the properties of individual team members and therefore subspace identification is established under an independent GP population. Teaming implies that the members of the resulting ensemble of classifiers should have explicitly non-overlapping behaviour. Performance evaluation is conducted over data sets taken from the UCI repository with 649–102,660 attributes and 2–10 classes. The resulting teams identify attribute spaces 1–4 orders of magnitude smaller than under the original data set. Moreover, team members generally consist of less than 10 instructions; thus, small attribute subspaces are not being traded for opaque models.     

Hierarchical multi-label classification using local neural networks

Hierarchical multi-label classification is a complex classification task where the classes involved in the problem are hierarchically structured and each example may simultaneously belong to more than one class in each hierarchical level. In this paper, we extend our previous works, where we investigated a new local-based classification method that incrementally trains a multi-layer perceptron for each level of the classification hierarchy. Predictions made by a neural network in a given level are used as inputs to the neural network responsible for the prediction in the next level. We compare the proposed method with one state-of-the-art decision-tree induction method and two decision-tree induction methods, using several hierarchical multi-label classification datasets. We perform a thorough experimental analysis, showing that our method obtains competitive results to a robust global method regarding both precision and recall evaluation measures.     

An economical cognitive approach for bi-objective optimization using bliss points, visualization, and interaction

When an optimization problem encompasses multiple objectives, it is usually difficult to define a single optimal solution. The decision maker plays an important role when choosing the final single decision. Pareto-based evolutionary multiobjective optimization (EMO) methods are very informative for the decision making process since they provide the decision maker with a set of efficient solutions to choose from. Despite that the set of efficient solutions may not be the global efficient set, we show in this paper that the set can still be informative when used in an interactive session with the decision maker. We use a combination of EMO and single objective optimization methods to guide the decision maker in interactive sessions.     

An effective and interpretable method for document classification

As the number of documents has been rapidly increasing in recent time, automatic text categorization is becoming a more important and fundamental task in information retrieval and text mining. Accuracy and interpretability are two important aspects of a text classifier. While the accuracy of a classifier measures the ability to correctly classify unseen data, interpretability is the ability of the classifier to be understood by humans and provide reasons why each data instance is assigned to a label. This paper proposes an interpretable classification method by exploiting the Dirichlet process mixture model of von Mises–Fisher distributions for directional data. By using the labeled information of the training data explicitly and determining automatically the number of topics for each class, the learned topics are coherent, relevant and discriminative. They help interpret as well as distinguish classes. Our experimental results showed the advantages of our approach in terms of separability, interpretability and effectiveness in classification task of datasets with high dimension and complex distribution. Our method is highly competitive with state-of-the-art approaches.     

A cooperative coevolutionary algorithm for instance selection for instance-based learning

This paper presents a cooperative evolutionary approach for the problem of instance selection for instance based learning. The model presented takes advantage of one of the recent paradigms in the field of evolutionary computation: cooperative coevolution. This paradigm is based on a similar approach to the philosophy of divide and conquer. In our method, the training set is divided into several subsets that are searched independently. A population of global solutions relates the search in different subsets and keeps track of the best combinations obtained. The proposed model has the advantage over standard methods in that it does not rely on any specific distance metric or classifier algorithm. Additionally, the fitness function of the individuals considers both storage requirements and classification accuracy, and the user can balance both objectives depending on his/her specific needs, assigning different weights to each one of these two terms. The method also shows good scalability when applied to large datasets. The proposed model is favorably compared with some of the most successful standard algorithms, IB3, ICF and DROP3, with a genetic algorithm using CHC method, and with four recent methods of instance selection, MSS, entropy-based instance selection, IMOEA and LVQPRU. The comparison shows a clear advantage of the proposed algorithm in terms of storage requirements, and is, at least, as good as any of the other methods in terms of testing error. A large set of 50 problems from the UCI Machine Learning Repository is used for the comparison. Additionally, a study of the effect of instance label noise is carried out, showing the robustness of the proposed algorithm.     

New methods for competitive coevolution

We consider "competitive coevolution," in which fitness is based on direct competition among individuals selected from two independently evolving populations of "hosts" and "parasites." Competitive coevolution can lead to an "arms race," in which the two populations reciprocally drive one another to increasing levels of performance and complexity. We use the games of Nim and 3-D Tic-Tac-Toe as test problems to explore three new techniques in competitive coevolution. "Competitive fitness sharing" changes the way fitness is measured; "shared sampling" provides a method for selecting a strong, diverse set of parasites; and the "hall of fame" encourages arms races by saving good individuals from prior generations. We provide several different motivations for these methods and mathematical insights into their use. Experimental comparisons are done, and a detailed analysis of these experiments is presented in terms of testing issues, diversity, extinction, arms race progress measurements, and drift.     

Cooperative bare-bone particle swarm optimization for data clustering

Cooperative coevolution (CC) was used to improve the performance of evolutionary algorithms (EAs) on complex optimization problems in a divide-and-conquer way. In this paper, we show that the CC framework can be very helpful to improve the performance of particle swarm optimization (PSO) on clustering high-dimensional datasets. Based on CC framework, the original partitional clustering problem is first decomposed to several subproblems, each of which is then evolved by an optimizer independently. We employ a very simple but efficient optimization algorithm, namely bare-bone particle swarm optimization (BPSO), as the optimizer to solve each subproblem cooperatively. In addition, we design a new centroid-based encoding schema for each particle and apply the Chernoff bounds to decide a proper population size. The experimental results on synthetic and real-life datasets illustrate the effectiveness and efficiency of the BPSO and CC framework. The comparisons show the proposed algorithm significantly outperforms five EA-based clustering algorithms, i.e., PSO, SRPSO, ACO, ABC and DE, and K-means on most of the datasets.     

一种基于混合粒子群优化算法的深度卷积神经网络架构搜索方法

神经架构搜索（neural architecture search,NAS）技术自动寻找神经网络中各层的最佳组合和连接方式,以及各种超参数的最佳分布。该方法从搜索空间生成若干不同的卷积神经网络（CNN）,使用混合粒子群优化（hybrid particle swarm optimization,HPSO）算法,将一定数目的神经网络个体视做一个群体,将每个网络个体在评价指标下的表现值视做适应度,在给定的世代数范围内,每个神经网络个体都学习自身的历史最佳适应度个体,和整个群体的最佳适应度个体,迭代改善自身的网络架构。实验结果表明,算法运行中出现的最优网络架构,在图像分类任务的多个基准数据集上,与手工设计的神经网络和以遗传算法为基础的NAS算法相比,在网络参数数量和准确率的平衡上取得了有竞争力的结果。