Continuous Lunches Are Free Plus the Design of Optimal Optimization Algorithms

This paper analyses extensions of No-Free-Lunch (NFL) theorems to countably infinite and uncountable infinite domains and investigates the design of optimal optimization algorithms. The original NFL theorem due to Wolpert and Macready states that, for finite search domains, all search heuristics have the same performance when averaged over the uniform distribution over all possible functions. For infinite domains the extension of the concept of distribution over all possible functions involves measurability issues and stochastic process theory. For countably infinite domains, we prove that the natural extension of NFL theorems, for the current formalization of probability, does not hold, but that a weaker form of NFL does hold, by stating the existence of non-trivial distributions of fitness leading to equal performances for all search heuristics. Our main result is that for continuous domains, NFL does not hold. This free-lunch theorem is based on the formalization of the concept of random fitness functions by means of random fields.     

Ranking-Based Black-Box Complexity

Randomized search heuristics such as evolutionary algorithms, simulated annealing, and ant colony optimization are a broadly used class of general-purpose algorithms. Analyzing them via classical methods of theoretical computer science is a growing field. While several strong runtime analysis results have appeared in the last 20 years, a powerful complexity theory for such algorithms is yet to be developed. We enrich the existing notions of black-box complexity by the additional restriction that not the actual objective values, but only the relative quality of the previously evaluated solutions may be taken into account by the black-box algorithm. Many randomized search heuristics belong to this class of algorithms. We show that the new ranking-based model can give more realistic complexity estimates. The class of all binary-value functions has a black-box complexity of O(logn) in the previous black-box models, but has a ranking-based complexity of Θ(n). On the other hand, for the class of all OneMax functions, we present a ranking-based black-box algorithm that has a runtime of Θ(n/logn), which shows that the OneMax problem does not become harder with the additional ranking-basedness restriction.     

基于预设时间收敛的分布式优化算法

考虑一类分布式优化问题,其目标是通过局部信息交互,使得局部成本函数之和构成的全局成本函数最小.针对该类问题,通过引入时基发生器(TBG),提出两种基于预设时间收敛的分布式比例积分(PI)优化算法.与现有的基于有限/固定时间收敛的分布式优化算法相比,所提出算法的收敛时间不依赖于系统的初值和参数,且可以任意预先设计.此外,在全局成本函数关于最优值点有限强凸,局部成本函数为可微的凸函数,且具有局部Lipschitz梯度的条件下,通过Lyapunov理论证明了所提算法都能实现预设时间收敛.最后,通过数值仿真验证了所提出算法的有效性.     

An energy function for the random neural network

Since Hopfield's seminal work on energy functions for neural networks and their consequence for the approximate solution of optimization problems, much attention has been devoted to neural heuristics for combinatorial optimization. These heuristics are often very time-consuming because of the need for randomization or Monte Carlo simulation during the search for solutions. In this paper, we propose a general energy function for a new neural model, the random neural model of Gelenbe. This model proposes a scheme of interaction between the neurons and not a dynamic equation of the system. Then, we apply this general energy function to different optimization problems.     

Non-smooth and non-convex problems in discrete optimal control

A class of discrete control problems described only by locally Lipschitz functions is studied from the point of view of necessary optimality conditions. Moreover, it is assumed that state and control constraints are given implicitly as general sets being approximated by the respective ‘ generalized ’ tangent or normal cone. To investigate these types of non-differentiable optimization problems some basic facts of the so called non-smooth analysis have to be applied. The crucial role is played by a ‘ generalized ’ gradient of a locally Lipschitz function. Using these concepts one is able to formulate necessary optimality conditions in a fairly general setting. However, to obtain necessary conditions in a more familiar form an alternative definition of a partial generalized gradient ia explored. Some special cases of discrete control problems are studied separately. In addition, also the maximum principle formulation of necessary conditions is investigated and a question of possible extensions is briefly discussed     

Upper and Lower Bounds for Randomized Search Heuristics in Black-Box Optimization

Randomized search heuristics like local search, tabu search, simulated annealing, or all kinds of evolutionary algorithms have many applications. However, for most problems the best worst-case expected run times are achieved by more problem-specific algorithms. This raises the question about the limits of general randomized search heuristics. Here a framework called black-box optimization is developed. The essential issue is that the problem but not the problem instance is knownto the algorithm which can collect information about the instance only by asking for the value of points in the search space. All known randomized search heuristics fit into this scenario. Lower bounds on the black-box complexity of problems are derived without complexity theoretical assumptions and are compared with upper bounds in this scenario.     

Complexity of pattern classes and the Lipschitz property

Rademacher and Gaussian complexities are successfully used in learning theory for measuring the capacity of the class of functions to be learnt. One of the most important properties for these complexities is their Lipschitz property: a composition of a class of functions with a fixed Lipschitz function may increase its complexity by at most twice the Lipschitz constant. The proof of this property is non-trivial (in contrast to the case for the other properties) and it is believed that the proof in the Gaussian case is conceptually more difficult than the one for the Rademacher case. In this paper we give a detailed proof of the Lipschitz property for the general case of a symmetric complexity measure that includes the Rademacher and Gaussian complexities as special cases.     

An Efficient Hybrid Particle Swarm Optimization Algorithm for Solving the Uncapacitated Continuous Location-Allocation Problem

Location-allocation problems are a class of complicated optimization problems that determine the location of facilities and the allocation of customers to the facilities. In this paper, the uncapacitated continuous location-allocation problem is considered, and a particle swarm optimization approach, which has not previously been applied to this problem, is presented. Two algorithms including classical and hybrid particle swarm optimization algorithms are developed. Local optima of the problem are obtained by two local search heuristics that exist in the literature. These algorithms are combined with particle swarm optimization to construct an efficient hybrid approach. Many large-scale problems are used to measure the effectiveness and efficiency of the proposed algorithms. Our results are compared with the best algorithms in the literature. The experimental results show that the hybrid PSO produces good solutions, is more efficient than the classical PSO, and is competitive with the best results from the literature. Additionally, the proposed hybrid PSO found better solutions for some instances than did the best known solutions in the literature.     

An adaptive single-point algorithm for global numerical optimization

This paper describes a novel algorithm for numerical optimization, called Simple Adaptive Climbing (SAC). SAC is a simple efficient single-point approach that does not require a careful fine-tunning of its two parameters. SAC algorithm shares many similarities with local optimization heuristics, such as random walk, gradient descent, and hill-climbing. SAC has a restarting mechanism, and a powerful adaptive mutation process that resembles the one used in Differential Evolution. The algorithms SAC is capable of performing global unconstrained optimization efficiently in high dimensional test functions. This paper shows results on 15 well-known unconstrained problems. Test results confirm that SAC is competitive against state-of-the-art approaches such as micro-Particle Swarm Optimization, CMA-ES or Simple Adaptive Differential Evolution.     

Evolutionary Algorithms for Quantum Computers

In this article, we formulate and study quantum analogues of randomized search heuristics, which make use of Grover search (in Proceedings of the 28th Annual ACM Symposium on Theory of Computing, pp. 212–219. ACM, New York, 1996) to accelerate the search for improved offsprings. We then specialize the above formulation to two specific search heuristics: Random Local Search and the (1+1) Evolutionary Algorithm. We call the resulting quantum versions of these search heuristics Quantum Local Search and the (1+1) Quantum Evolutionary Algorithm. We conduct a rigorous runtime analysis of these quantum search heuristics in the computation model of quantum algorithms, which, besides classical computation steps, also permits those unique to quantum computing devices. To this end, we study the six elementary pseudo-Boolean optimization problems OneMax, LeadingOnes, Discrepancy, Needle, Jump, and TinyTrap. It turns out that the advantage of the respective quantum search heuristic over its classical counterpart varies with the problem structure and ranges from no speedup at all for the problem Discrepancy to exponential speedup for the problem TinyTrap. We show that these runtime behaviors are closely linked to the probabilities of performing successful mutations in the classical algorithms.     

Adaptive estimation of component faults and actuator faults for nonlinear one‐sided Lipschitz systems

Fault estimation for classical nonlinear Lipschitz systems has been subject to several research works. So far, much less interest has been given to the more generalized class of systems, namely, one‐sided Lipchitz systems. Dealing with component faults and actuator faults, only very few works have been done to reconstruct these types of faults for this new class of systems. A major limitation of the previous works is that the fault vector to be estimated there does not give any information about the actual faulty physical parameters of the system, so component faults and actuator faults are not distinguishable. In this paper, a set of possible faulty parameters in the system is estimated. Component faults and actuator faults are separated and distinguished. The effectiveness of the proposed method is shown through simulations for a numerical example.     

On the investigation of hyper-heuristics on a financial forecasting problem

Financial forecasting is a really important area in computational finance, with numerous works in the literature. This importance can be reflected in the literature by the continuous development of new algorithms. Hyper-heuristics have been successfully used in the past for a number of search and optimization problems, and have shown very promising results. To the best of our knowledge, they have not been used for financial forecasting. In this paper we present pioneer work, where we use different hyper-heuristics frameworks to investigate whether we can improve the performance of a financial forecasting tool called EDDIE 8. EDDIE 8 allows the GP (Genetic Programming) to search in the search space of indicators for solutions, instead of using pre-specified ones; as a result, its search area has dramatically increased and sometimes solutions can be missed due to ineffective search. We apply 14 different low-level heuristics to EDDIE 8, to 30 different datasets, and examine their effect to the algorithm’s performance. We then select the most prominent heuristics and combine them into three different hyper-heuristics frameworks. Results show that all three frameworks are competitive, and are able to show significantly improved results, especially in the case of best results. Lastly, analysis on the weights of the heuristics shows that there can be a constant swinging among some of the low-level heuristics, which denotes that the hyper-heuristics frameworks are able to ‘know’ the appropriate time to switch from one heuristic to the other, based on their effectiveness.     

Energy performance of heuristics and meta-heuristics for real-time joint resource scaling and consolidation in virtualized networked data centers

In this paper, we explore on a comparative basis the performance suitability of meta-heuristic, sometime denoted as random search algorithms, and greedy-type heuristics for the energy-saving joint dynamic scaling and consolidation of the network-plus-computing resources hosted by networked virtualized data centers when the target is the support of real-time streaming-type applications. For this purpose, the energy and delay performances of Tabu Search (TS), Simulated Annealing (SA) and Evolutionary Strategy (ES) meta-heuristics are tested and compared with the corresponding ones of Best-Fit Decreasing-type heuristics, in order to give insight on the resulting performance-versus-implementation complexity trade-offs. In principle, the considered meta-heuristics and heuristics are general formal approaches that can be applied to large classes of (typically, non-convex and mixed integer) optimization problems. However, specially for the meta-heuristics, a main challenge is to design them to properly address the real-time joint computing-plus-networking resource consolidation and scaling optimization problem. To this purpose, the aim of this paper is: (i) introduce a novel Virtual Machine Allocation (VMA) scheme that aims at choosing a suitable set of possible Virtual Machine placements among the (possibly, non-homogeneous) set of available servers; (ii) propose a new class of random search algorithms (RSAs) denoted as consolidation meta-heuristic, considering the VMA problem in RSAs. In particular, the design of novel variants of meta-heuristics, namely TS-RSC, SA-RSC and ES-RSC, is particularized to the resource scaling and consolidation (RSC) problem; (iii) compare the results of the obtained new RSAs class against some state-of-the-art heuristic approaches. A set of experimental results, both simulated and real-world ones, support the effectiveness of the proposed approaches against the traditional ones.     

Randomized local search,evolutionary algorithms,and the minimum spanning tree problem

Randomized search heuristics, among them randomized local search and evolutionary algorithms, are applied to problems whose structure is not well understood, as well as to problems in combinatorial optimization. The analysis of these randomized search heuristics has been started for some well-known problems, and this approach is followed here for the minimum spanning tree problem. After motivating this line of research, it is shown that randomized search heuristics find minimum spanning trees in expected polynomial time without employing the global technique of greedy algorithms.     

GRMA: Generalized Range Move Algorithms for the Efficient Optimization of MRFs

Markov random fields (MRF) have become an important tool for many vision applications, and the optimization of MRFs is a problem of fundamental importance. Recently, Veksler and Kumar et al. proposed the range move algorithms, which are some of the most successful optimizers. Instead of considering only two labels as in previous move-making algorithms, they explore a large search space over a range of labels in each iteration, and significantly outperform previous move-making algorithms. However, two problems have greatly limited the applicability of range move algorithms: (1) They are limited in the energy functions they can handle (i.e., only truncated convex functions); (2) They tend to be very slow compared to other move-making algorithms (e.g., \(\alpha \)-expansion and \(\alpha \beta \)-swap). In this paper, we propose two generalized range move algorithms (GRMA) for the efficient optimization of MRFs. To address the first problem, we extend the GRMAs to more general energy functions by restricting the chosen labels in each move so that the energy function is submodular on the chosen subset. Furthermore, we provide a feasible sufficient condition for choosing these subsets of labels. To address the second problem, we dynamically obtain the iterative moves by solving set cover problems. This greatly reduces the number of moves during the optimization. We also propose a fast graph construction method for the GRMAs. Experiments show that the GRMAs offer a great speedup over previous range move algorithms, while yielding competitive solutions.     

Functions represented by BL-algebra formulas: characterization and approximate representation

The paper is a contribution to the study of functional systems of BL-algebras on [0,1] extended by constants from [0,1] and suprema and infima over (infinite) sets of different instances of the same formula. We prove that the class of functions which are represented by thus specified formulas coincides with the class of extensional functions. Moreover, the class of these functions coincides with the class of Lipschitz continuous functions in the case of BL-algebra whose t-norm has a continuous additive generator. We also consider an approximate representation of functions, represented by BL-formulas with a possibly infinite length, by discrete normal forms and suggest an algorithmic approach to its construction. This paper has been partially supported by grant IAA1187301 of the GA AV ČR.     

Rethinking the design of real-coded evolutionary algorithms: Making discrete choices in continuous search domains

Although real-coded evolutionary algorithms (EAs) have been applied to optimization problems for over thirty years, the convergence properties of these methods remain poorly understood. We discuss the use of discrete random variables to perform search in real-valued EAs. Although most real-valued EAs perform mutation with continuous random variables, we argue that EAs using discrete random variables for mutation can be much easier to analyze. In particular, we present and analyze two simple EAs that make discrete choices of mutation steps.     

基于事件触发的分布式优化算法

本文研究了一类分布式优化问题,其目标是通过局部信息交换使由局部成本函数之和构成的全局成本函数最小.针对无向连通图,我们提出了两种基于比例积分策略的分布式优化算法.在局部成本函数可微且凸的条件下,证明了所提算法渐近收敛到全局最小值点.更进一步,在局部成本函数具有局部Lipschitz梯度和全局成本函数关于全局最小值点是有...     

Obstacle avoidance extremum seeking control based on constrained derivative-free optimization

A new control scheme based on extremum seeking control (ESC) which employs a constrained derivative-free optimization algorithm has been proposed in this paper. A theorem has been formulated to prove the convergence result of ESC based on constrained derivative-free optimization. Generalized pattern search method with filter algorithm for constraint is used to generate a sequence of ESC control state. Since generalized pattern search (GPS) method does not require continuously differentiable and Lipschitz conditions, noise cancellation algorithm is added to the proposed ESC algorithm which is then used for multi-agent robot system. The obstacles are expressed as constraint functions instead of the traditional way of calculating the performance function of obstacles. Simulation results illustrate a multi-agent obstacle avoidance system which utilized the control algorithm to avoid obstacles that appear on the path of multi-agent robots. Based on the simulation results, it can be observed that multi-agents maintain their formation as per initial condition and follow the target without colliding into obstacles while navigating in a noisy environment. Performance comparison of the proposed algorithm with a reference algorithm shows the efficiency of the proposed algorithm.     

Notes on averaging over acyclic digraphs and discrete coverage control

In this paper we study averaging algorithms and coverage control laws in a unified light. First, we characterize the convergence properties of averaging algorithms over acyclic digraphs with fixed and controlled-switching topology. Second, we introduce and study novel discrete coverage control laws, which are useful in practical implementations of coverage strategies. We characterize the close relationship of the novel discrete control laws with continuous coverage control laws and with averaging algorithms over a class of acyclic digraphs, that we term discrete Voronoi graphs. These results provide a unified framework to model a vast class of distributed optimization problems.