A comprehensive survey on optimizing deep learning models by metaheuristics

Deep neural networks (DNNs), which are extensions of artificial neural networks, can learn higher levels of feature hierarchy established by lower level features by transforming the raw feature space to another complex feature space. Although deep networks are successful in a wide range of problems in different fields, there are some issues affecting their overall performance such as selecting appropriate values for model parameters, deciding the optimal architecture and feature representation and determining optimal weight and bias values. Recently, metaheuristic algorithms have been proposed to automate these tasks. This survey gives brief information about common basic DNN architectures including convolutional neural networks, unsupervised pre-trained models, recurrent neural networks and recursive neural networks. We formulate the optimization problems in DNN design such as architecture optimization, hyper-parameter optimization, training and feature representation level optimization. The encoding schemes used in metaheuristics to represent the network architectures are categorized. The evolutionary and selection operators, and also speed-up methods are summarized, and the main approaches to validate the results of networks designed by metaheuristics are provided. Moreover, we group the studies on the metaheuristics for deep neural networks based on the problem type considered and present the datasets mostly used in the studies for the readers. We discuss about the pros and cons of utilizing metaheuristics in deep learning field and give some future directions for connecting the metaheuristics and deep learning. To the best of our knowledge, this is the most comprehensive survey about metaheuristics used in deep learning field.

     

Multi‐objective optimization using metaheuristics: non‐standard algorithms

In recent years, the application of metaheuristic techniques to solve multi‐objective optimization problems has become an active research area. Solving this kind of problems involves obtaining a set of Pareto‐optimal solutions in such a way that the corresponding Pareto front fulfils the requirements of convergence to the true Pareto front and uniform diversity. Most of the studies on metaheuristics for multi‐objective optimization are focused on Evolutionary Algorithms, and some of the state‐of‐the‐art techniques belong this class of algorithms. Our goal in this paper is to study open research lines related to metaheuristics but focusing on less explored areas to provide new perspectives to those researchers interested in multi‐objective optimization. In particular, we focus on non‐evolutionary metaheuristics, hybrid multi‐objective metaheuristics, parallel multi‐objective optimization and multi‐objective optimization under uncertainty. We analyze these issues and discuss open research lines.     

A survey on metaheuristics for optimization in food manufacturing industry

This paper surveys recent articles on the applications of metaheuristics for solving optimization problems in the food manufacturing industry. Metaheuristics for decision making has attracted significant research and industry attention due to the increasing complexity of models and quick decision making requirements in the industry. Metaheuristics have been applied to food processing/production technologies including fermentation, thermal drying and distillation and other system wide optimization such as transportation, storage (warehousing), production planning and scheduling. In terms of metaheuristics algorithms, Genetic Algorithm and Differential Evolution are the most popular while other algorithms have also demonstrated their effectiveness in addressing various optimization problems. Most problems were typically formulated as single objective mathematical models constructed from experimental or collected data. Recently, multi-objective optimization is becoming more popular because it is able to consider problems from several perspectives and attain more practical results.     

An ant colony optimization algorithm for DNA sequencing by hybridization

The reconstruction of DNA sequences from DNA fragments is one of the most challenging problems in computational biology. In recent years the specific problem of DNA sequencing by hybridization has attracted quite a lot of interest in the optimization community. Several metaheuristics such as tabu search and evolutionary algorithms have been applied to this problem. However, the performance of existing metaheuristics is often inferior to the performance of recently proposed constructive heuristics. On the basis of these new heuristics we develop an ant colony optimization algorithm for DNA sequencing by hybridization. An important feature of this algorithm is the implementation in a so-called multi-level framework. The computational results show that our algorithm is currently a state-of-the-art method for the tackled problem.     

N-Body Simulation Inspired by Metaheuristics Optimization

The N-body problem in classical physics, is the calculation of force of gravitational attraction of heavenly bodies towards each other. Solving this problem for many heavenly bodies has always posed a challenge to physicists and mathematicians. Large number of bodies, huge masses, long distances and exponentially increasing number of equations of motion of the bodies have been the major hurdles in solving this problem for large and complex galaxies. Advent of high performance computational machines have mitigated the problem to much extent, but still for large number of bodies it consumes huge amount of resources and days for computation. Conventional algorithms have been able to reduce the computational complexity from to by splitting the space into a tree or mesh network, researchers are still looking for improvements. In this research work we propose a novel solution to N-body problem inspired by metaheuristics algorithms. The proposed algorithm is simulated for various time periods of selected heavenly bodies and analyzed for speed and accuracy. The results are compared with that of conventional algorithms. The outcomes show about 50% time saving with almost no loss in accuracy. The proposed approach being a metaheuristics optimization technique, attempts to find optimal solution to the problem, searching the entire space in a unique and efficient manner in a very limited amount of time.     

Fast Parallel Algorithms for a Broad Class of Nonlinear Variational Diffusion Approaches

Variational segmentation and nonlinear diffusion approaches have been very active research areas in the fields of image processing and computer vision during recent years. In the present paper, we review recent advances in the development of efficient numerical algorithms for these approaches. The performance of parallel implementations of these algorithms on general-purpose hardware is assessed. A mathematically clear connection between variational models and nonlinear diffusion filters is presented that allows to interpret one approach as an approximation of the other, and vice versa. Extending this continuous connection to the fully discrete setting enables us to derive many structural similarities between efficient numerical algorithms for both frameworks. These results provide a perspective for uniform implementations of nonlinear variational models and diffusion filters on parallel architectures.     

Hybrid of genetic algorithm and local search to solve MAX-SAT problem using nVidia CUDA framework

General Purpose computing over Graphical Processing Units (GPGPUs) is a huge shift of paradigm in parallel computing that promises a dramatic increase in performance. But GPGPUs also bring an unprecedented level of complexity in algorithmic design and software development. In this paper we describe the challenges and design choices involved in parallelizing a hybrid of Genetic Algorithm (GA) and Local Search (LS) to solve MAXimum SATisfiability (MAX-SAT) problem on a state-of-the-art nVidia Tesla GPU using nVidia Compute Unified Device Architecture (CUDA). MAX-SAT is a problem of practical importance and is often solved by employing metaheuristics based search methods like GAs and hybrid of GA with LS. Almost all the parallel GAs (pGAs) designed in the last two decades were designed for either clusters or MPPs. Unfortunately, very little research is done on the implementation of such algorithms over commodity graphics hardware. GAs in their simple form are not suitable for implementation over the Single Instruction Multiple Thread (SIMT) architecture of a GPU, and the same is the case with conventional LS algorithms. In this paper we explore different genetic operators that can be used for an efficient implementation of GAs over nVidia GPUs. We also design and introduce new techniques/operators for an efficient implementation of GAs and LS over such architectures. We use nVidia Tesla C1060 to perform several numerical tests and performance measurements and show that in the best case we obtain a speedup of 25×. We also discuss the effects of different optimization techniques on the overall execution time.     

A Novel State Transition Method for Metaheuristic-Based Scheduling in Heterogeneous Computing Systems

Much of the recent literature shows a prevalance in the use of metaheuristics in solving a variety of problems in parallel and distributed computing. This is especially ture for problems that have a combinatorial nature, such as scheduling and load balancing. Despite numerous efforts, task scheduling remains one of the most challenging problems in heterogeneous computing environments. In this paper, we propose a new state transitionscheme , called the Duplication-based State Transition (DST) method specially designed for metaheuristics that can be used for the task scheduling problem in heterogeneous computing environments. State transition in metaheuristics is a key component that takes charge of generating variants of a given state. The DST method produces a new state by first overlapping randomly generated states with the current state and then the resultant state is refined by removing ineffectual tasks. The proposed method is incorporated into three different metaheuristics: genetic algorithms (GAs), simulated annealing (SA), and artificial immune system (AISs). They are experimentally evaluated and are also compared with existing algorithms. The experimental results confirm DST's promising impact on the performance of metaheuristics.     

Water wave optimization for combinatorial optimization: Design strategies and applications

Although the community of nature-inspired computing has witnessed a wide variety of metaheuristics, it often requires considerable effort to adapt them to different combinatorial optimization problems (COPs), and few studies have been devoted to reducing this burden. This paper proposes a systematic approach that consists of a set of basic steps and strategies for adapting water wave optimization (WWO), a simple and generic metaheuristic, to concrete heuristic algorithms for different COPs. Taking advantages of the generic algorithmic framework, designers can only focus on adapting the prorogation operator and the wavelength calculation method according to the combinatorial properties of the given problem, and thus easily derive efficient problem-solving algorithms. We illustrate and test our approach on the flow-shop scheduling problem (FSP), the single-objective multidimensional knapsack problem (MKP), and the multi-objective MKP, and then present an application to a machine utilization optimization problem for a large manufacturing enterprise. The results demonstrate that our approach can derive concrete algorithms that are competitive to the state-of-the-arts. Our approach also provides insights into the adaptation of other metaheuristics and the development of new metaheuristics for COPs.     

Parallel architectures and algorithms for image component labeling

A survey and a characterization of the various parallel algorithms and architectures developed for the problem of labeling digitized images over the last two decades are presented. It is shown that four basic parallel techniques underly the various parallel algorithms for this problem. However, because most of these techniques have been developed at a theoretical level, it is still not clear which techniques are most efficient in practical terms. Parallel architectures and parallel models of computation that implement these techniques are also studied     

Investigating Multi-View Differential Evolution for solving constrained engineering design problems

Several constrained and unconstrained optimization problems have been adequately solved over the years thanks to advances in the metaheuristics area. In the last decades, different metaheuristics have been proposed employing new ideas, and hybrid algorithms that improve the original metaheuristics have been developed. One of the most successfully employed metaheuristics is the Differential Evolution. In this paper it is proposed a Multi-View Differential Evolution algorithm (MVDE) in which several mutation strategies are applied to the current population to generate different views at each iteration. The views are then merged according to the winner-takes-all paradigm, resulting in automatic exploration/exploitation balance. MVDE was tested to solve a set of well-known constrained engineering design problems and the obtained results were compared to those from many state-of-the-art metaheuristics. Results show that MVDE was very competitive in the considered problems, largely outperforming several of the compared algorithms.     

同时送取货车辆路径问题算法研究综述

针对同时送取货车辆路径问题的研究算法进行了评述.将该问题的求解方法分为精确算法、构造型启发式、现代启发式以及并行算法四个大类.从算法的原理、性能、适用环境,以及算法之间差异性等方面对各类算法进行了较为全面的介绍.最后,说明了VRPSDP算法研究在节点具有双重需求车辆路径问题理论研究方面的意义,并提出未来VRPSDP算法研究的两个发展方向,即适合多处理器上运行的并行现代启发式算法,以及有效的混合算法如量子行为粒子群算法.     

Ensemble of multi-objective metaheuristic algorithms for multi-objective unconstrained binary quadratic programming problem

Metaheuristics have been widely utilized for solving NP-hard optimization problems. However, these algorithms usually perform differently from one problem to another, i.e., one may be effective on a problem but performs badly on another problem. Therefore, it is difficult to choose the best algorithm in advance for a given problem. In contrast to selecting the best algorithm for a problem, selection hyper-heuristics aim at performing well on a set of problems (instances). This paper proposes a selection hyper-heuristic based algorithm for multi-objective optimization problems. In the proposed algorithm, multiple metaheuristics exhibiting different search behaviors are managed and controlled as low-level metaheuristics in an algorithm pool, and the most appropriate metaheuristic is selected by means of a performance indicator at each search stage. To assess the performance of the proposed algorithm, an implementation of the algorithm containing four metaheuristics is proposed and tested for solving multi-objective unconstrained binary quadratic programming problem. Experimental results on 50 benchmark instances show that the proposed algorithm can provide better overall performance than single metaheuristics, which demonstrates the effectiveness of the proposed algorithm.     

并行智能优化算法研究进展

基于种群迭代搜索的智能优化算法在农业、交通、工业等很多领域都取得了广泛的应用.但是该类算法迭代寻优的特点使其求解效率通常较低,很难应用到大规模、高维或实时性要求较高的复杂优化问题中.随并行分布式技术的发展,国内外很多学者开始着手研究智能优化算法的并行化.本文首要介绍了并行智能优化算法的基本概念;其次从协同机制、并行模型以及硬件结构3个维度综述了几类常见的并行智能优化算法,详细分析阐述了它们优点及不足;最后对并行智能优化算法的未来研究进行了展望.     

Optimization of one-dimensional Bin Packing Problem with island parallel grouping genetic algorithms

The well-known one-dimensional Bin Packing Problem (BPP) of whose variants arise in many real life situations is a challenging NP-Hard combinatorial optimization problem. Metaheuristics are widely used optimization tools to find (near-) optimal solutions for solving large problem instances of BPP in reasonable running times. With this study, we propose a set of robust and scalable hybrid parallel algorithms that take advantage of parallel computation techniques, evolutionary grouping genetic metaheuristics, and bin-oriented heuristics to obtain solutions for large scale one-dimensional BPP instances. A total number of 1318 benchmark problems are examined with the proposed algorithms and it is shown that optimal solutions for 88.5% of these instances can be obtained with practical optimization times while solving the rest of the problems with no more than one extra bin. When the results are compared with the existing state-of-the-art heuristics, the developed parallel hybrid grouping genetic algorithms can be considered as one of the best one-dimensional BPP algorithms in terms of computation time and solution quality.     

Development and analysis of cooperative model-based metaheuristics

The paper proposes a methodology to construct cooperative metaheuristic methods for solving combinatorial optimization problems using model-based algorithms. Its distinctive feature is that the original problem is solved by a search (optimization) in the space of models. Such a search is performed on the basis of models formed by basic algorithms. Cooperative metaheuristics underlain by ant colony optimization and MH-method algorithms are developed, and the efficiency of the proposed methodology is evaluated by means of a computational experiment.     

Automatic clustering using nature-inspired metaheuristics: A survey

In cluster analysis, a fundamental problem is to determine the best estimate of the number of clusters; this is known as the automatic clustering problem. Because of lack of prior domain knowledge, it is difficult to choose an appropriate number of clusters, especially when the data have many dimensions, when clusters differ widely in shape, size, and density, and when overlapping exists among groups. In the late 1990s, the automatic clustering problem gave rise to a new era in cluster analysis with the application of nature-inspired metaheuristics. Since then, researchers have developed several new algorithms in this field. This paper presents an up-to-date review of all major nature-inspired metaheuristic algorithms used thus far for automatic clustering. Also, the main components involved during the formulation of metaheuristics for automatic clustering are presented, such as encoding schemes, validity indices, and proximity measures. A total of 65 automatic clustering approaches are reviewed, which are based on single-solution, single-objective, and multiobjective metaheuristics, whose usage percentages are 3%, 69%, and 28%, respectively. Single-objective clustering algorithms are adequate to efficiently group linearly separable clusters. However, a strong tendency in using multiobjective algorithms is found nowadays to address non-linearly separable problems. Finally, a discussion and some emerging research directions are presented.     

On Enforced Convergence of ACO and its Implementation on the Reconfigurable Mesh Architecture Using Size Reduction Tasks

In this paper we show that size reduction tasks can be used for executing iterative randomized metaheuristics on runtime reconfigurable architectures so that an improved throughput and better solution qualities are obtained compared to conventional architectures that do not allow runtime reconfiguration. In particular, the problem of executing ant colony optimization (ACO) algorithms on a dynamically reconfigurable mesh architecture is studied. It is shown how ACO can be implemented such that the convergence behavior of the algorithm can be used to dynamically reduce the size of the submesh that is needed for execution. Furthermore we propose a method to enforce the convergence of ACO leading to a faster reduction process. This increases the throughput of ACO algorithms on runtime reconfigurable meshes. The increased throughput is used for repeated runs of ACO algorithms on a given set of problem instances which significantly improves the obtained solution quality.     

Parallel custom instruction identification for extensible processors

With the ability of customization for an application domain, extensible processors have been used more and more in embedded systems in recent years. Extensible processors customize an application domain by executing parts of application code in hardware instead of software. Determining parts of application code as custom instruction generally requires subgraph enumeration and subgraph selection. Both subgraph enumeration problem and subgraph selection problem are computationally difficult problems. Most of previous works focus on sequential algorithms for these two problems. In this paper, we present a parallel implementation of a latest subgraph enumeration algorithm based on a computer cluster. A standard ant colony optimization algorithm (ACO), a modified version of ACO with local optimum search and a parallel ACO algorithm are also proposed to solve the subgraph selection problem in this work. Experimental results show that the parallel algorithms outperform the sequential algorithms in terms of runtime or (and) quality of results. In addition, we have formally proved the upper bound on the number of feasible solutions in subgraph selection problem with or without the overlapping constraint.