Speeding up backpropagation using multiobjective evolutionary algorithms

The use of backpropagation for training artificial neural networks (ANNs) is usually associated with a long training process. The user needs to experiment with a number of network architectures; with larger networks, more computational cost in terms of training time is required. The objective of this letter is to present an optimization algorithm, comprising a multiobjective evolutionary algorithm and a gradient-based local search. In the rest of the letter, this is referred to as the memetic Pareto artificial neural network algorithm for training ANNs. The evolutionary approach is used to train the network and simultaneously optimize its architecture. The result is a set of networks, with each network in the set attempting to optimize both the training error and the architecture. We also present a self-adaptive version with lower computational cost. We show empirically that the proposed method is capable of reducing the training time compared to gradient-based techniques.     

Performance evaluation of evolutionary multiobjective optimization algorithms for multiobjective fuzzy genetics-based machine learning

Recently, evolutionary multiobjective optimization (EMO) algorithms have been utilized for the design of accurate and interpretable fuzzy rule-based systems. This research area is often referred to as multiobjective genetic fuzzy systems (MoGFS), where EMO algorithms are used to search for non-dominated fuzzy rule-based systems with respect to their accuracy and interpretability. In this paper, we examine the ability of EMO algorithms to efficiently search for Pareto optimal or near Pareto optimal fuzzy rule-based systems for classification problems. We use NSGA-II (elitist non-dominated sorting genetic algorithm), its variants, and MOEA/D (multiobjective evolutionary algorithm based on decomposition) in our multiobjective fuzzy genetics-based machine learning (MoFGBML) algorithm. Classification performance of obtained fuzzy rule-based systems by each EMO algorithm is evaluated for training data and test data under various settings of the available computation load and the granularity of fuzzy partitions. Experimental results in this paper suggest that reported classification performance of MoGFS in the literature can be further improved using more computation load, more efficient EMO algorithms, and/or more antecedent fuzzy sets from finer fuzzy partitions.     

Closed-loop evolutionary multiobjective optimization

Artificial evolution has been used for more than 50 years as a method of optimization in engineering, operations research and computational intelligence. In closed-loop evolution (a term used by the statistician, George Box) or, equivalently, evolutionary experimentation (Ingo Rechenberg?s terminology), the ?phenotypes? are evaluated in the real world by conducting a physical experiment, whilst selection and breeding is simulated. Well-known early work on artificial evolution?design engineering problems in fluid dynamics, and chemical plant process optimization? was carried out in this experimental mode. More recently, the closed-loop approach has been successfully used in much evolvable hardware and evolutionary robotics research, and in some microbiology and biochemistry applications. In this article, several further new targets for closed-loop evolutionary and multiobjective optimization are considered. Four case studies from my own collaborative work are described: (i) instrument optimization in analytical biochemistry; (ii) finding effective drug combinations in vitro; (iii) onchip synthetic biomolecule design; and (iv) improving chocolate production processes. Accurate simulation in these applications is not possible due to complexity or a lack of adequate analytical models. In these and other applications discussed, optimizing experimentally brings with it several challenges: noise; nuisance factors; ephemeral resource constraints; expensive evaluations, and evaluations that must be done in (large) batches. Evolutionary algorithms (EAs) are largely equal to these vagaries, whilst modern multiobjective EAs also enable tradeoffs among conflicting optimization goals to be explored. Nevertheless, principles from other disciplines, such as statistics, Design of Experiments, machine learning and global optimization are also relevant to aspects of the closed-loop problem, and may inspire futher development of multiobjective EAs.     

Mean-semivariance portfolio optimization with multiobjective evolutionary algorithms and technical analysis rules

Recent work has been devoted to study the use of multiobjective evolutionary algorithms (MOEAs) in stock portfolio optimization, within a common mean-variance framework. This article proposes the use of a more appropriate framework, mean-semivariance framework, which takes into account only adverse return variations instead of overall variations. It also proposes the use and comparison of established technical analysis (TA) indicators in pursuing better outcomes within the risk-return relation. Results show there is some difference in the performance of the two selected MOEAs – non-dominated sorting genetic algorithm II (NSGA II) and strength pareto evolutionary algorithm 2 (SPEA 2) – within portfolio optimization. In addition, when used with four TA based strategies – relative strength index (RSI), moving average convergence/divergence (MACD), contrarian bollinger bands (CBB) and bollinger bands (BB), the two selected MOEAs achieve solutions with interesting in-sample and out-of-sample outcomes for the BB strategy.     

A tool for multiobjective evolutionary algorithms

This paper introduces a software tool based on illustrative applications for the development, analysis and application of multiobjective evolutionary algorithms. The multiobjective evolutionary algorithms tool (MOEAT) written in C# using a variety of multiobjective evolutionary algorithms (MOEAs) offers a powerful environment for various kinds of optimization tasks. It has many useful features such as visualizing of the progress and the results of optimization in a dynamic or static mode, and decision variable settings. The performance measurements of well-known multiobjective evolutionary algorithms in MOEAT are done using benchmark problems. In addition, two case studies from engineering domain are presented.     

Integrated multiobjective optimization and a priori preferences using genetic algorithms

One of the tasks of decision-making support systems is to develop methods that help the designer select a solution among a set of actions, e.g. by constructing a function expressing his/her preferences over a set of potential solutions. In this paper, a new method to solve multiobjective optimization (MOO) problems is developed in which the user’s information about his/her preferences is taken into account within the search process. Preference functions are built that reflect the decision-maker’s (DM) interests and use meaningful parameters for each objective. The preference functions convert these objective preferences into numbers. Next, a single objective is automatically built and no weight selection is performed. Problems found due to the multimodality nature of a generated single cost index are managed with Genetic Algorithms (GAs). Three examples are given to illustrate the effectiveness of the method.     

Target-matching test problem for multiobjective topology optimization using genetic algorithms

This paper describes the multiobjective topology optimization of continuum structures solved as a discrete optimization problem using a multiobjective genetic algorithm (GA) with proficient constraint handling. Crucial to the effectiveness of the methodology is the use of a morphological geometry representation that defines valid structural geometries that are inherently free from checkerboard patterns, disconnected segments, or poor connectivity. A graph- theoretic chromosome encoding, together with compatible reproduction operators, helps facilitate the transmission of topological/shape characteristics across generations in the evolutionary process. A multicriterion target-matching problem developed here is a novel test problem, where a predefined target geometry is the known optimum solution, and the good results obtained in solving this problem provide a convincing demonstration and a quantitative measure of how close to the true optimum the solutions achieved by GA methods can be. The methodology is then used to successfully design a path-generating compliant mechanism by solving a multicriterion structural topology optimization problem.     

Comparison of multiobjective evolutionary algorithms: empirical results

In this paper, we provide a systematic comparison of various evolutionary approaches to multiobjective optimization using six carefully chosen test functions. Each test function involves a particular feature that is known to cause difficulty in the evolutionary optimization process, mainly in converging to the Pareto-optimal front (e.g., multimodality and deception). By investigating these different problem features separately, it is possible to predict the kind of problems to which a certain technique is or is not well suited. However, in contrast to what was suspected beforehand, the experimental results indicate a hierarchy of the algorithms under consideration. Furthermore, the emerging effects are evidence that the suggested test functions provide sufficient complexity to compare multiobjective optimizers. Finally, elitism is shown to be an important factor for improving evolutionary multiobjective search.     

Optimization of interval type-2 fuzzy logic controllers using evolutionary algorithms

A method for designing optimal interval type-2 fuzzy logic controllers using evolutionary algorithms is presented in this paper. Interval type-2 fuzzy controllers can outperform conventional type-1 fuzzy controllers when the problem has a high degree of uncertainty. However, designing interval type-2 fuzzy controllers is more difficult because there are more parameters involved. In this paper, interval type-2 fuzzy systems are approximated with the average of two type-1 fuzzy systems, which has been shown to give good results in control if the type-1 fuzzy systems can be obtained appropriately. An evolutionary algorithm is applied to find the optimal interval type-2 fuzzy system as mentioned above. The human evolutionary model is applied for optimizing the interval type-2 fuzzy controller for a particular non-linear plant and results are compared against an optimal type-1 fuzzy controller. A comparative study of simulation results of the type-2 and type-1 fuzzy controllers, under different noise levels, is also presented. Simulation results show that interval type-2 fuzzy controllers obtained with the evolutionary algorithm outperform type-1 fuzzy controllers.     

A diversity preserving selection in multiobjective evolutionary algorithms

In this paper, an efficient diversity preserving selection (DPS) technique is presented for multiobjective evolutionary algorithms (MEAs). The main goal is to preserve diversity of nondominated solutions in problems with scaled objectives. This is achieved with the help of a mechanism that preserves certain inferior individuals over successive generations with a view to provide long term advantages. The mechanism selects a group (of individuals) that is statistically furthest from the worst group, instead of just concentrating on the best individuals, as in truncation selection. In a way, DPS judiciously combines the diversity preserving mechanism with conventional truncation selection. Experiments demonstrate that DPS significantly improves diversity of nondominated solutions in badly-scaling problems, while at the same time it exhibits acceptable proximity performance. Whilst DPS has certain advantages when it comes to scaling problems, it empirically shows no disadvantages for the problems with non-scaled objectives.     

Optimization of retaining wall design using evolutionary algorithms

This paper explores the performance of three evolutionary optimization methods, differential evolution (DE), evolutionary strategy (ES) and biogeography based optimization algorithm (BBO), for nonlinear constrained optimum design of a cantilever retaining wall. These algorithms are based on biological contests for survival and reproduction. The retaining wall optimization problem consists of two criteria, geotechnical stability and structural strength, while the final design minimizes an objective function. The objective function is defined in terms of both cost and weight. Constraints are applied using the penalty function method. The efficiency of the proposed method is examined by means of two numerical retaining wall design examples, one with a base shear key and one without a base shear key. The final designs are compared to the ones determined by genetic algorithms as classical metaheuristic optimization methods. The design results and convergence rate of the BBO algorithm show a significantly better performance than the other algorithms in both design cases.     

A multiobjective evolutionary algorithm toolbox for computer-aidedmultiobjective optimization

This paper presents an interactive graphical user interface (GUI) based multiobjective evolutionary algorithm (MOEA) toolbox for effective computer-aided multiobjective (MO) optimization. Without the need of aggregating multiple criteria into a compromise function, it incorporates the concept of Pareto's optimality to evolve a family of nondominated solutions distributing along the tradeoffs uniformly. The toolbox is also designed with many useful features such as the goal and priority settings to provide better support for decision-making in MO optimization, dynamic population size that is computed adaptively according to the online discovered Pareto-front, soft/hard goal settings for constraint handlings, multiple goals specification for logical "AND"/"OR" operation, adaptive niching scheme for uniform population distribution, and a useful convergence representation for MO optimization. The MOEA toolbox is freely available for download at http://vlab.ee.nus.edu.sg/-kctan/moea.htm which is ready for immediate use with minimal knowledge needed in evolutionary computing. To use the toolbox, the user merely needs to provide a simple "model" file that specifies the objective function corresponding to his/her particular optimization problem. Other aspects like decision variable settings, optimization process monitoring and graphical results analysis can be performed easily through the embedded GUIs in the toolbox. The effectiveness and applications of the toolbox are illustrated via the design optimization problem of a practical ill-conditioned distillation system. Performance of the algorithm in MOEA toolbox is also compared with other well-known evolutionary MO optimization methods upon a benchmark problem.     

Combining convergence and diversity in evolutionary multiobjective optimization

Over the past few years, the research on evolutionary algorithms has demonstrated their niche in solving multiobjective optimization problems, where the goal is to find a number of Pareto-optimal solutions in a single simulation run. Many studies have depicted different ways evolutionary algorithms can progress towards the Pareto-optimal set with a widely spread distribution of solutions. However, none of the multiobjective evolutionary algorithms (MOEAs) has a proof of convergence to the true Pareto-optimal solutions with a wide diversity among the solutions. In this paper, we discuss why a number of earlier MOEAs do not have such properties. Based on the concept of epsilon-dominance, new archiving strategies are proposed that overcome this fundamental problem and provably lead to MOEAs that have both the desired convergence and distribution properties. A number of modifications to the baseline algorithm are also suggested. The concept of epsilon-dominance introduced in this paper is practical and should make the proposed algorithms useful to researchers and practitioners alike.     

On improving multiobjective genetic algorithms for design optimization

This paper presents some improvements to Multi-Objective Genetic Algorithms (MOGAs). MOGA modifies certain operators within the GA itself to produce a multiobjective optimization technique. The improvements are made to overcome some of the shortcomings in niche formation, stopping criteria and interaction with a design decision-maker. The technique involves filtering, mating restrictions, the idea of objective constraints, and detecting Pareto solutions in the non-convex region of the Pareto set. A step-by-step procedure for an improved MOGA has been developed and demonstrated via two multiobjective engineering design examples: (i) two-bar truss design, and (ii) vibrating platform design. The two-bar truss example has continuous variables while the vibrating platform example has mixed-discrete (combinatorial) variables. Both examples are solved by MOGA with and without the improvements. It is shown that MOGA with the improvements performs better for both examples in terms of the number of function evaluations.     

Convergence analysis of some multiobjective evolutionary algorithms when discovering motifs

An important issue in multiobjective optimization is the study of the convergence speed of algorithms. An optimization problem must be defined as simple as possible to minimize the computational cost required to solve it. In this work, we study the convergence speed of seven multiobjective evolutionary algorithms: DEPT, MO-VNS, MOABC, MO-GSA, MO-FA, NSGA-II, and SPEA2; when solving an important biological problem: the motif discovery problem. We have used twelve instances of four different organisms as benchmark, analyzing the number of fitness function evaluations required by each algorithm to achieve reasonable quality solutions. We have used the hypervolume indicator to evaluate the solutions discovered by each algorithm, measuring its quality every 100 evaluations. This methodology also allows us to study the hit rates of the algorithms over 30 independent runs. Moreover, we have made a deeper study in the more complex instance of each organism. In this study, we observe the increase of the archive (number of non-dominated solutions) and the spread of the Pareto fronts obtained by the algorithm in the median execution. As we will see, our study reveals that DEPT, MOABC, and MO-FA provide the best convergence speeds and the highest hit rates.     

Running time analysis of multiobjective evolutionary algorithms on pseudo-Boolean functions

This paper presents a rigorous running time analysis of evolutionary algorithms on pseudo-Boolean multiobjective optimization problems. We propose and analyze different population-based algorithms, the simple evolutionary multiobjective optimizer (SEMO), and two improved versions, fair evolutionary multiobjective optimizer (FEMO) and greedy evolutionary multiobjective optimizer (GEMO). The analysis is carried out on two biobjective model problems, leading ones trailing zeroes (LOTZ) and count ones count zeroes (COCZ), as well as on the scalable m-objective versions mLOTZ and mCOCZ. Results on the running time of the different population-based algorithms and for an alternative approach, a multistart (1+1)-EA based on the /spl epsi/-constraint method, are derived. The comparison reveals that for many problems, the simple algorithm SEMO is as efficient as this (1+1)-EA. For some problems, the improved variants FEMO and GEMO are provably better. For the analysis, we propose and apply two general tools, an upper bound technique based on a decision space partition and a randomized graph search algorithm, which facilitate the analysis considerably.     

The balance between proximity and diversity in multiobjective evolutionary algorithms

Over the last decade, a variety of evolutionary algorithms (EAs) have been proposed for solving multiobjective optimization problems. Especially more recent multiobjective evolutionary algorithms (MOEAs) have been shown to be efficient and superior to earlier approaches. An important question however is whether we can expect such improvements to converge onto a specific efficient MOEA that behaves best on a large variety of problems. In this paper, we argue that the development of new MOEAs cannot converge onto a single new most efficient MOEA because the performance of MOEAs shows characteristics of multiobjective problems. While we point out the most important aspects for designing competent MOEAs in this paper, we also indicate the inherent multiobjective tradeoff in multiobjective optimization between proximity and diversity preservation. We discuss the impact of this tradeoff on the concepts and design of exploration and exploitation operators. We also present a general framework for competent MOEAs and show how current state-of-the-art MOEAs can be obtained by making choices within this framework. Furthermore, we show an example of how we can separate nondomination selection pressure from diversity preservation selection pressure and discuss the impact of changing the ratio between these components.