Hyperparameter self-tuning for data streams

The number of Internet of Things devices generating data streams is expected to grow exponentially with the support of emergent technologies such as 5G networks. Therefore, the online processing of these data streams requires the design and development of suitable machine learning algorithms, able to learn online, as data is generated. Like their batch-learning counterparts, stream-based learning algorithms require careful hyperparameter settings. However, this problem is exacerbated in online learning settings, especially with the occurrence of concept drifts, which frequently require the reconfiguration of hyperparameters. In this article, we present SSPT, an extension of the Self Parameter Tuning (SPT) optimisation algorithm for data streams. We apply the Nelder–Mead algorithm to dynamically-sized samples, converging to optimal settings in a single pass over data while using a relatively small number of hyperparameter configurations. In addition, our proposal automatically readjusts hyperparameters when concept drift occurs. To assess the effectiveness of SSPT, the algorithm is evaluated with three different machine learning problems: recommendation, regression, and classification. Experiments with well-known data sets show that the proposed algorithm can outperform previous hyperparameter tuning efforts by human experts. Results also show that SSPT converges significantly faster and presents at least similar accuracy when compared with the previous double-pass version of the SPT algorithm.     

Properties of an adaptive archiving algorithm for storing nondominated vectors

Search algorithms for Pareto optimization are designed to obtain multiple solutions, each offering a different trade-off of the problem objectives. To make the different solutions available at the end of an algorithm run, procedures are needed for storing them, one by one, as they are found. In a simple case, this may be achieved by placing each point that is found into an "archive" which maintains only nondominated points and discards all others. However, even a set of mutually nondominated points is potentially very large, necessitating a bound on the archive's capacity. But with such a bound in place, it is no longer obvious which points should be maintained and which discarded; we would like the archive to maintain a representative and well-distributed subset of the points generated by the search algorithm, and also that this set converges. To achieve these objectives, we propose an adaptive archiving algorithm, suitable for use with any Pareto optimization algorithm, which has various useful properties as follows. It maintains an archive of bounded size, encourages an even distribution of points across the Pareto front, is computationally efficient, and we are able to prove a form of convergence. The method proposed here maintains evenness, efficiency, and cardinality, and provably converges under certain conditions but not all. Finally, the notions underlying our convergence proofs support a new way to rigorously define what is meant by "good spread of points" across a Pareto front, in the context of grid-based archiving schemes. This leads to proofs and conjectures applicable to archive sizing and grid sizing in any Pareto optimization algorithm maintaining a grid-based archive.     

Classification rule mining using ant programming guided by grammar with multiple Pareto fronts

This paper proposes a multi-objective ant programming algorithm for mining classification rules, MOGBAP, which focuses on optimizing sensitivity, specificity, and comprehensibility. It defines a context-free grammar that restricts the search space and ensures the creation of valid individuals, and its heuristic function presents two complementary components. Moreover, the algorithm addresses the classification problem from a new multi-objective perspective specifically suited for this task, which finds an independent Pareto front of individuals per class, so that it avoids the overlapping problem that appears when measuring the fitness of individuals from different classes. A comparative analysis of MOGBAP using two and three objectives is performed, and then its performance is experimentally evaluated throughout 15 varied benchmark data sets and compared to those obtained using another eight relevant rule extraction algorithms. The results prove that MOGBAP outperforms the other algorithms in predictive accuracy, also achieving a good trade-off between accuracy and comprehensibility.     

Performance evaluation of evolutionary multiobjective optimization algorithms for multiobjective fuzzy genetics-based machine learning

Recently, evolutionary multiobjective optimization (EMO) algorithms have been utilized for the design of accurate and interpretable fuzzy rule-based systems. This research area is often referred to as multiobjective genetic fuzzy systems (MoGFS), where EMO algorithms are used to search for non-dominated fuzzy rule-based systems with respect to their accuracy and interpretability. In this paper, we examine the ability of EMO algorithms to efficiently search for Pareto optimal or near Pareto optimal fuzzy rule-based systems for classification problems. We use NSGA-II (elitist non-dominated sorting genetic algorithm), its variants, and MOEA/D (multiobjective evolutionary algorithm based on decomposition) in our multiobjective fuzzy genetics-based machine learning (MoFGBML) algorithm. Classification performance of obtained fuzzy rule-based systems by each EMO algorithm is evaluated for training data and test data under various settings of the available computation load and the granularity of fuzzy partitions. Experimental results in this paper suggest that reported classification performance of MoGFS in the literature can be further improved using more computation load, more efficient EMO algorithms, and/or more antecedent fuzzy sets from finer fuzzy partitions.     

Extraction of decision alternatives in construction management projects: Application and adaptation of NSGA-II and MOPSO

The time-cost trade-off problem is a known bi-objective problem in the field of project management. Recently, a new parameter, the quality of the project has been added to previously considered time and cost parameters. The main specification of the time-cost trade-off problem is discretization of the decision space to limited and accountable decision variables. In this situation the efficiency of the traditional methods decrease and applying of the evolutionary algorithms is necessary. In this paper, two evolutionary algorithms that originally search the decision space in a continuous manner including: (1) multi-objective particle swarm optimization (MOPSO) and (2) nondominated sorting genetic algorithm (NSGA)-II, are considered as the optimization tools to solve two construction project management problems. These problems are both in discrete domain including two or tree objectives, separately. In this regard, some procedures has been suggested and then applied to adopt both algorithms capable in solving the problems in a discrete domain. Results show the advantages and effectiveness of the used procedures in reporting the optimal Pareto for the optimization problems. Moreover, the NSGA-II is more successful in determining optimal alternatives in both time-cost trade-off (TCTO) and time-cost-quality trade-off (TCQTO) problems than the MOPSO algorithm.     

A New Solution Path Algorithm in Support Vector Regression

In this paper, regularization path algorithms were proposed as a novel approach to the model selection problem by exploring the path of possibly all solutions with respect to some regularization hyperparameter in an efficient way. This approach was later extended to a support vector regression (SVR) model called epsiv -SVR. However, the method requires that the error parameter epsiv be set a priori. This is only possible if the desired accuracy of the approximation can be specified in advance. In this paper, we analyze the solution space for epsiv-SVR and propose a new solution path algorithm, called epsiv-path algorithm, which traces the solution path with respect to the hyperparameter epsiv rather than lambda. Although both two solution path algorithms possess the desirable piecewise linearity property, our epsiv-path algorithm overcomes some limitations of the original lambda-path algorithm and has more advantages. It is thus more appealing for practical use.     

A new and fast rival genetic algorithm for feature selection

Feature selection is one of the significant steps in classification tasks. It is a pre-processing step to select a small subset of significant features that can contribute the most to the classification process. Presently, many metaheuristic optimization algorithms were successfully applied for feature selection. The genetic algorithm (GA) as a fundamental optimization tool has been widely used in feature selection tasks. However, GA suffers from the hyperparameter setting, high computational complexity, and the randomness of selection operation. Therefore, we propose a new rival genetic algorithm, as well as a fast version of rival genetic algorithm, to enhance the performance of GA in feature selection. The proposed approaches utilize the competition strategy that combines the new selection and crossover schemes, which aim to improve the global search capability. Moreover, a dynamic mutation rate is proposed to enhance the search behaviour of the algorithm in the mutation process. The proposed approaches are validated on 23 benchmark datasets collected from the UCI machine learning repository and Arizona State University. In comparison with other competitors, proposed approach can provide highly competing results and overtake other algorithms in feature selection.

     

多目标协调进化算法研究

进化算法适合解决多目标优化问题,但难以产生高维优化问题的最优解,文中针对此问题提出了一种求解高维目标优化问题的新进化方法,即多目标协调进化算法,主要特点是进化群体按协调模型使用偏好信息进行偏好排序,而不是基于Pareto优于关系进行了个体排序,实验结果表明,所提出的算法是可行而有效的,且能在有限进化代数内收敛。     

Game model-based co-evolutionary algorithm with non-dominated memory and Euclidean distance selection mechanisms for multi-objective optimization

Many real-world problems involve simultaneous optimization of several incommensurable and often competing objectives. In the search for solutions to multi-objective optimization problems (MOPs), we find that there is no single optimum but rather a set of optimums known as the “Pareto optimal set”. Co-evolutionary algorithms are well suited to optimization problems which involve several often competing objectives. Co-evolutionary algorithms are aimed at evolving individuals through individuals competing in an objective space. In order to approximate the ideal Pareto optimal set, the search capability of diverse individuals in an objective space can be used to determine the performance of evolutionary algorithms. Non-dominated memory and Euclidean distance selection mechanisms for co-evolutionary algorithms have the goal of overcoming the limited search capability of diverse individuals in the population space. In this paper, we propose a method for maintaining population diversity in game model-based co-evolutionary algorithms, and we evaluate the effectiveness of our approach by comparing it with other methods through rigorous experiments on several MOPs.     

Evolutionary multi-objective optimisation with preferences for multivariable PI controller tuning

Multi-objective optimisation design procedures have shown to be a valuable tool for control engineers. They enable the designer having a close embedment of the tuning process for a wide variety of applications. In such procedures, evolutionary multi-objective optimisation has been extensively used for PI and PID controller tuning; one reason for this is due to their flexibility to include mechanisms in order to enhance convergence and diversity. Although its usability, when dealing with multi-variable processes, the resulting Pareto front approximation might not be useful, due to the number of design objectives stated. That is, a vast region of the objective space might be impractical or useless a priori, due to the strong degradation in some of the design objectives. In this paper preference handling techniques are incorporated into the optimisation process, seeking to improve the pertinency of the approximated Pareto front for multi-variable PI controller tuning. That is, the inclusion of preferences into the optimisation process, in order to seek actively for a pertinent Pareto front approximation. With such approach, it is possible to tune a multi-variable PI controller, fulfilling several design objectives, using previous knowledge from the designer on the expected trade-off performance. This is validated with a well-known benchmark example in multi-variable control. Control tests show the usefulness of the proposed approach when compared with other tuning techniques.     

Ordered subsets Bayesian tomographic reconstruction using 2-D smoothing splines as priors

The ordered subsets expectation maximization (OS-EM) algorithm has enjoyed considerable interest for accelerating the well-known EM algorithm for emission tomography. The OS principle has also been applied to several regularized EM algorithms, such as nonquadratic convex minimization-based maximum a posteriori (MAP) algorithms. However, most of these methods have not been as practical as OS-EM due to their complex optimization methods and difficulties in hyperparameter estimation. We note here that, by relaxing the requirement of imposing sharp edges and using instead useful quadratic spline priors, solutions are much easier to compute, and hyperparameter calculation becomes less of a problem. In this work, we use two-dimensional smoothing splines as priors and apply a method of iterated conditional modes for the optimization. In this case, step sizes or line-search algorithms necessary for gradient-based descent methods are avoided. We also accelerate the resulting algorithm using the OS approach and propose a principled way of scaling smoothing parameters to retain the strength of smoothing for different subset numbers. Our experimental results show that the OS approach applied to our quadratic MAP algorithms provides a considerable acceleration while retaining the advantages of quadratic spline priors.     

Pareto optimal mapping for tile-based network-on-chip under reliability constraints

Mapping for network-on-chip (NoC) is one of the key steps of NoC design. To improve the performance and reliability of NoC architectures, we present a comprehensive optimization algorithm with multiple objectives. We propose to find the Pareto optimal solutions, rather than a single solution usually obtained through scalarization, e.g. weighting the objective functions. In order to meet the NoC mapping requests and strengthen the capability of searching solutions, the standard particle swarm optimization (PSO) algorithm is improved and a fault-tolerant routing is proposed. These methods help to solve the tradeoff between high performance and system reliability. We present a mathematical analysis of the convergence of the improved algorithms, and prove sufficient conditions of convergence. The improved algorithms are implemented on the Embedded Systems Synthesis Benchmarks Suite (E3S). Experimental results show our algorithms achieve high performance and reliability compared with the standard PSO.     

基于alpha支配的高维目标进化算法研究

基于Pareto支配的多目标进化算法能够很好地处理2～3维的多目标优化问题。但在处理高维多目标问题时,随着目标维数的增大,支配受阻解的数量急剧增加,导致现有的多目标算法存在选择压力不够、优化效果较差的问题。通过引入α支配提供严格的Pareto分层,在同层中挑选相对稀疏的解作为候选解,同时详细分析不同α对算法性能的影响,提出一种新的基于α偏序和拥塞距离抽样的高维目标进化算法。将该算法在DTLZ上进行性能测试,并采用世代距离(GD)、空间评价(SP)、超体积(HV)等多个指标评估算法的性能。实验结果表明,引入α支配能去除绝大部分支配受阻解(DRSs),提高算法的收敛性。与快速非支配排序算法(NSGA-II)、基于分解的多目标进化算法(MOEA/D)、基于距离更新的分解多目标进化算法(MOEA/D-DU)相比,该算法的整体解集的质量 有明显提高。     

Multi-objective evolutionary optimization of polynomial neural networks for modelling and prediction of explosive cutting process

In this paper, evolutionary algorithms (EAs) are deployed for multi-objective Pareto optimal design of group method of data handling (GMDH)-type neural networks which have been used for modelling an explosive cutting process using some input–output experimental data. In this way, multi-objective EAs (non-dominated sorting genetic algorithm, NSGA-II) with a new diversity-preserving mechanism are used for Pareto optimization of such GMDH-type neural networks. The important conflicting objectives of GMDH-type neural networks that are considered in this work are, namely, training error (TE), prediction error (PE), and number of neurons (N) of such neural networks. Different pairs of theses objective functions are selected for 2-objective optimization processes. Therefore, optimal Pareto fronts of such models are obtained in each case which exhibit the trade-off between the corresponding pair of conflicting objectives and, thus, provide different non-dominated optimal choices of GMDH-type neural networks models for explosive cutting process. Moreover, all the three objectives are considered in a 3-objective optimization process, which consequently leads to some more non-dominated choices of GMDH-type models representing the trade-offs among the training error, prediction error, and number of neurons (complexity of network), simultaneously. The overlay graphs of these Pareto fronts also reveal that the 3-objective results include those of the 2-objective results and, thus, provide more optimal choices for the multi-objective design of GMDH-type neural networks in terms of minimum training error, minimum prediction error, and minimum complexity.     

Searching for knee regions of the Pareto front using mobile reference points

Evolutionary Algorithms (EAs) have been recognized to be well suited to approximate the Pareto front of Multi-objective Optimization Problems (MOPs). In reality, the Decision Maker (DM) is not interested in discovering the whole Pareto front rather than finding only the portion(s) of the front that matches at most his/her preferences. Recently, several studies have addressed the decision-making task to assist the DM in choosing the final alternative. Knee regions are potential parts of the Pareto front presenting the maximal trade-offs between objectives. Solutions residing in knee regions are characterized by the fact that a small improvement in either objective will cause a large deterioration in at least another one which makes moving in either direction not attractive. Thus, in the absence of explicit DM’s preferences, we suppose that knee regions represent the DM’s preferences themselves. Recently, few works were proposed to find knee regions. This paper represents a further study in this direction. Hence, we propose a new evolutionary method, denoted TKR-NSGA-II, to discover knee regions of the Pareto front. In this method, the population is guided gradually by means of a set of mobile reference points. Since the reference points are updated based on trade-off information, the population converges towards knee region centers which allows the construction of a neighborhood of solutions in each knee. The performance assessment of the proposed algorithm is done on two- and three-objective knee-based test problems. The obtained results show the ability of the algorithm to: (1) find the Pareto optimal knee regions, (2) control the extent (We mean by extent the breadth/spread of the obtained knee region.) of the obtained regions independently of the geometry of the front and (3) provide competitive and better results when compared to other recently proposed methods. Moreover, we propose an interactive version of TKR-NSGA-II which is useful when the DM has no a priori information about the number of existing knees in the Pareto optimal front.     

云计算环境中面向DAG任务的多目标调度算法

为了实现任务执行效率与执行代价的同步优化,提出了一种云计算环境中的DAG任务多目标调度优化算法。算法将多目标最优化问题以满足Pareto最优的均衡最优解集合的形式进行建模,以启发式方式对模型进行求解;同时,为了衡量多目标均衡解的质量,设计了基于hypervolume方法的评估机制,从而可以得到相互冲突目标间的均衡调度解。通过配置云环境与三种人工合成工作流和两种现实科学工作流的仿真实验测试,结果表明,比较同类单目标算法和多目标启发式算法,算法不仅求解质量更高,而且解的均衡度更好,更加符合现实云的资源使用特征与工作流调度模式。     

Bicriteria scheduling of a material handling robot in an m-machine cell to minimize the energy consumption of the robot and the cycle time

This study considers a flowshop type production system consisting of m machines. A material handling robot transports the parts between the machines and loads and unloads the machines. We consider the sequencing of the robot moves and determining the speeds of these moves simultaneously. These decisions affect both the robot’s energy consumption and the production speed of the system. In this study, these two objectives are considered simultaneously. We propose a second order cone programming formulation to find Pareto efficient solutions. We also develop a heuristic algorithm that finds a set of approximate Pareto efficient solutions. The conic formulation can find robot schedules for small cells with less number of machines in reasonable computation times. Our heuristic algorithm can generate a large set of approximate Pareto efficient solutions in a very short computational time. Proposed solution approaches help the decision-maker to achieve the best trade-off between the throughput of a cell and the energy efficiency of a material handling robot.     

Model-based transductive learning of the kernel matrix

This paper addresses the problem of transductive learning of the kernel matrix from a probabilistic perspective. We define the kernel matrix as a Wishart process prior and construct a hierarchical generative model for kernel matrix learning. Specifically, we consider the target kernel matrix as a random matrix following the Wishart distribution with a positive definite parameter matrix and a degree of freedom. This parameter matrix, in turn, has the inverted Wishart distribution (with a positive definite hyperparameter matrix) as its conjugate prior and the degree of freedom is equal to the dimensionality of the feature space induced by the target kernel. Resorting to a missing data problem, we devise an expectation-maximization (EM) algorithm to infer the missing data, parameter matrix and feature dimensionality in a maximum a posteriori (MAP) manner. Using different settings for the target kernel and hyperparameter matrices, our model can be applied to different types of learning problems. In particular, we consider its application in a semi-supervised learning setting and present two classification methods. Classification experiments are reported on some benchmark data sets with encouraging results. In addition, we also devise the EM algorithm for kernel matrix completion. Editor: Philip M. Long     

Tuning of PID controller based on a multiobjective genetic algorithm applied to a robotic manipulator

Most controllers optimization and design problems are multiobjective in nature, since they normally have several (possibly conflicting) objectives that must be satisfied at the same time. Instead of aiming at finding a single solution, the multiobjective optimization methods try to produce a set of good trade-off solutions from which the decision maker may select one. Several methods have been devised for solving multiobjective optimization problems in control systems field. Traditionally, classical optimization algorithms based on nonlinear programming or optimal control theories are applied to obtain the solution of such problems. The presence of multiple objectives in a problem usually gives rise to a set of optimal solutions, largely known as Pareto-optimal solutions. Recently, Multiobjective Evolutionary Algorithms (MOEAs) have been applied to control systems problems. Compared with mathematical programming, MOEAs are very suitable to solve multiobjective optimization problems, because they deal simultaneously with a set of solutions and find a number of Pareto optimal solutions in a single run of algorithm. Starting from a set of initial solutions, MOEAs use iteratively improving optimization techniques to find the optimal solutions. In every iterative progress, MOEAs favor population-based Pareto dominance as a measure of fitness. In the MOEAs context, the Non-dominated Sorting Genetic Algorithm (NSGA-II) has been successfully applied to solving many multiobjective problems. This paper presents the design and the tuning of two PID (Proportional–Integral–Derivative) controllers through the NSGA-II approach. Simulation numerical results of multivariable PID control and convergence of the NSGA-II is presented and discussed with application in a robotic manipulator of two-degree-of-freedom. The proposed optimization method based on NSGA-II offers an effective way to implement simple but robust solutions providing a good reference tracking performance in closed loop.