An LP-based hyperparameter optimization model for language modeling

In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models’ hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most popular cost functions. In this study, we propose a fractional nonlinear programming model that finds the optimal perplexity value. The special structure of the model allows us to approximate it by a linear programming model that can be solved using the well-known simplex algorithm. To the best of our knowledge, this is the first attempt to use optimization techniques to find perplexity values in the language modeling literature. We apply our model to find hyperparameters of a language model and compare it to the grid search algorithm. Furthermore, we illustrate that it results in lower perplexity values. We perform this experiment on a real-world dataset from SwiftKey to validate our proposed approach.     

支持向量分类机在入侵检测中的应用研究

为解决入侵检测系统的泛化能力问题,分析了多类分类器的理论框架,并综合考虑训练集数据的预处理、交叉验证时间和入侵检测模型准确率三个因素,提出了一种改进的粗细网格参数优化算法。在基于支持向量机的入侵检测模型中,将KDD数据集映射到高维空间,并采用不同的算法对核函数相关参数进行优化。实例仿真计算表明,通过改进的网格搜索法所获得的参数相对来说有明显的时间优势,分类精度和效率得到了提高。     

Aligning block permutation methods for topology transformation on computational grids

Many traditional parallel matrix computing algorithms are performed on regular resource topologies, such as mesh. However, the grid resource topology is often irregular in practice. In this paper, we present a transformation algorithm of grid resource topology for achieving virtual meshes. And on the virtual mesh, these traditional parallel algorithms can be performed in a modern computational grid environment. The basic idea of our topology transformation is to align the basic blocks of grid computational resources through permutations in a virtual mesh. Designing a cost function of heuristic search scheme for the transformation, we use it to fully exploit the computational and communicational abilities of grid resources. The experiment results show that our aligning block permutation can significantly reduce the time complexity of search tree. They also show that the heuristic search scheme can effectively find the block permutation that makes better use of computational and communicational abilities of grid resources.     

Sequence-dependent group scheduling problem on unrelated-parallel machines

In this research we address a sequence-dependent group scheduling problem on a set of unrelated-parallel machines where the run time of each job differs on different machines. To benefit both producer and customers we attempt to minimize a linear combination of total weighted completion time and total weighted tardiness. Since the problem is shown to be NP-hard, meta-heuristic algorithms based on tabu search are developed to find the optimal/near optimal solution. For some small size yet complex problems, the results from these algorithms are compared to the optimal solutions found by CPLEX. The result obtained in all of these problems is that the tabu search algorithms could find solutions at least as good as CPLEX but in drastically shorter computational time, thus signifying the high degree of efficiency and efficacy attained by the former.     

求解0-1背包问题的混合贪婪遗传算法

求解0-1背包问题（KP）的最优解的时候,传统遗传算法（GA）的局部求精能力不足而简单局部搜索算法的全局探索能力有限,针对上述问题,将这两个算法整合并提出了混合贪婪遗传算法（HGGA）。在GA全局搜索框架下增加局部搜索模块,并改进传统仅基于物品价值密度的修复算子,增加基于物品价值的贪婪混合选项,从而加速寻优过程。HGGA一方面引导种群在进化的优质解空间中展开精细搜索,另一方面依靠GA的经典操作算子开拓全局搜索空间,从而达到算法求精能力和开拓能力的良好平衡。HGGA分别在三组数据上做了测试,结果表明在第一组15个测试用例中的12个上,HGGA能够百分百找到最优解,成功率达到80%;在第二组小规模数据集上,HGGA的性能明显好于其他同类GA和其他元启发算法;在第三组大规模数据集上,HGGA较其他元启发式算法具有更好的稳定性和高效性。     

New validation methods for improving standard and multi-parametric support vector regression training time

The selection of hyper-parameters in support vector regression algorithms (SVMr) is an essential process in the training of these learning machines. Unfortunately, there is not an exact method to obtain the optimal values of SVMr hyper-parameters. Therefore, it is necessary to use a search algorithm and sometimes a validation method in order to find the best combination of hyper-parameters. The problem is that the SVMr training time can be huge in large training databases if standard search algorithms and validation methods (such as grid search and K-fold cross validation), are used. In this paper we propose two novel validation methods which reduce the SVMr training time, maintaining the accuracy of the final machine. We show the good performance of both methods in the standard SVMr with 3 hyper-parameters (where the hyper-parameters search is usually carried out by means of a grid search) and also in the extension to multi-parametric kernels, where meta-heuristic approaches such as evolutionary algorithms must be used to look for the best set of SVMr hyper-parameters. In all cases the new validation methods have provided very good results in terms of training time, without affecting the final SVMr accuracy.     

一种基于网格方法的高维数据流子空间聚类算法

基于对网格聚类方法的分析,结合由底向上的网格方法和自顶向下的网格方法,设计了一个能在线处理高维数据流的子空间聚类算法。通过利用由底向上网格方法对数据的压缩能力和自顶向下网格方法处理高维数据的能力,算法能基于对数据流的一次扫描,快速识别数据中位于不同子空间内的簇。理论分析以及在多个数据集上的实验表明算法具有较高的计算精度与计算效率。     

Path-relinking Tabu search for the multi-objective flexible job shop scheduling problem

The multi-objective flexible job shop scheduling problem is solved using a novel path-relinking algorithm based on the state-of-the-art Tabu search algorithm with back-jump tracking. A routing solution is identified by problem-specific neighborhood search, and is then further refined by the Tabu search algorithm with back-jump tracking for a sequencing decision. The resultant solution is used to maintain the medium-term memory where the best solutions are stored. A path-relinking heuristics is designed to generate diverse solutions in the most promising areas. An improved version of the algorithm is then developed by incorporating an effective dimension-oriented intensification search to find solutions that are located near extreme solutions. The proposed algorithms are tested on benchmark instances and its experimental performance is compared with that of algorithms in the literature. Comparison results show that the proposed algorithms are competitive in terms of its computation performance and solution quality.     

一种改进的主题网络蜘蛛搜索算法

主题网络蜘蛛搜索策略是专业搜索引擎的核心技术。但是目前的主题搜索算法往往存在很大贪婪性,难以在全局范围内找到最优解。通过比较分析发现Best-First算法虽然有它的不足,但是它在几种算法中表现的性能最优。故以Best-First算法为基础,提出了BS-BS算法。对BS-BS算法进行性能评价,发现应用此算法搜索不但“召回率”有所提高,还能在一定程度上找到全局范围内的最优解。     

Optimization based on phylogram analysis

Estimation of Distribution Algorithms (EDAs) is evolutionary algorithms with relevant performance in handling complex problems. Nevertheless, their efficiency and effectiveness directly depends on how accurate the deployed probabilistic models are, which in turn depend on methods of model building. Although the best models found in the literature are often built by computationally complex methods, whose corresponding EDAs require high running time, these methods may evaluate a lesser number of points in the search space. In order to find a better trade-off between running time (efficiency) and the number of evaluated points (effectiveness), this work uses probabilistic models built by algorithms of phylogenetic reconstruction, since some of them are able to efficiently produce accurate models. Then, an EDA, namely, Optimization based on Phylogram Analysis, and a new search technique, namely, Composed Exhaustive Search, are developed and proposed to find solutions for combinatorial optimization problems with different levels of difficulty. Experimental results show that the proposed new EDA features an interesting trade-off between running time and number of evaluated points, attaining solutions near to the best results found in the literature for each one of such performance measures.     

A new effective operator for the hybrid algorithm for solving global optimisation problems

Hybrid algorithms have been recently used to solve complex single-objective optimisation problems. The ultimate goal is to find an optimised global solution by using these algorithms. Based on the existing algorithms (HP_CRO, PSO, RCCRO), this study proposes a new hybrid algorithm called MPC (Mean-PSO-CRO), which utilises a new Mean-Search Operator. By employing this new operator, the proposed algorithm improves the search ability on areas of the solution space that the other operators of previous algorithms do not explore. Specifically, the Mean-Search Operator helps find the better solutions in comparison with other algorithms. Moreover, the authors have proposed two parameters for balancing local and global search and between various types of local search, as well. In addition, three versions of this operator, which use different constraints, are introduced. The experimental results on 23 benchmark functions, which are used in previous works, show that our framework can find better optimal or close-to-optimal solutions with faster convergence speed for most of the benchmark functions, especially the high-dimensional functions. Thus, the proposed algorithm is more effective in solving single-objective optimisation problems than the other existing algorithms.     

On multiprocessor task scheduling using efficient state space search approaches

Obtaining an optimal schedule for a set of precedence-constrained tasks is a well-known NP-complete problem in its general form. In view of the intractability of the problem, most of the previous work relies on heuristics that try to find reasonably high quality solutions in an acceptable amount of time. While optimal polynomial-time algorithms are known only for a few simple cases (and in other cases can only be obtained through an exhaustive search with prohibitively high time complexity), they may be critically important for applications in which performance is the prime objective. Optimal solutions can also serve as a reference to test the performance of various heuristics. Moreover, an optimal schedule for a program at hand needs to be determined only once (and off-line) but the program using that schedule is in general executed several times. In this paper, we propose optimal algorithms for static scheduling of task graphs with arbitrary parameters to multiple homogeneous processors. The first algorithm is based on the A^* search technique and uses a computationally efficient cost function for guiding the search with reduced complexity. Additionally, we propose a number of effective state-pruning techniques to reduce the search space. For further lowering the complexity, we propose an efficient parallelization of the search algorithm. We parallelize the algorithm with reduced interprocessor communication as well as with static and dynamic load-balancing schemes to evenly distribute the search states to the processors. We also propose an approximate algorithm that guarantees a bounded deviation from the optimal solution but executes in a considerably shorter time. Based on an extensive experimental evaluation of the algorithms, we conclude that the parallel algorithm with pruning techniques is an efficient scheme for generating optimal solutions of reasonably large problems while the approximate algorithm is effective if slightly degraded solutions are acceptable.     

Local Search Algorithms for the Maximal Planar Layout Problem

The problem of laying out facilities is practically important in a modern manufacturing environment. This problem can be formulated as a weighted maximal planar graph in which vertices represent facilities and edge weights represent desirability measures between facilities. The objective is to find a planar graph that can be drawn on a plane without any edges intersecting with the highest sum of edge weights. Exact solution method can only solve small sized problems. In this paper, local search algorithms based on steepest ascent, hybrid simulated annealing and tabu search with a non-monotonic cooling schedule, and tabu search with a hashing function are developed to obtain near-optimal solutions. Different search strategies are investigated. All the developed algorithms are compared with existing construction methods and a branch and bound exact algorithm on a set of practical size problems. The proposed algorithms performed very well in terms of solution quality and computation time.     

融合分区和局部搜索的多模态多目标优化

为解决多模态多目标优化中种群多样性维持难和所得等价解数量不足问题,基于分区搜索和局部搜索,本研究提出一种融合分区和局部搜索的多模态多目标粒子群算法(multimodal multi-objective particle swarm optimization combing zoning search and local search,ZLS-SMPSO-MM)。在所提算法中,整个搜索空间被分割成多个子空间以维持种群多样性和降低搜索难度;然后,使用已有的自组织多模态多目标粒子群算法在每个子空间搜索等价解和挖掘邻域信息,并利用局部搜索能力较强的协方差矩阵自适应算法对有潜力的区域进行精细搜索。通过14个多模态多目标优化问题测试,并与其他5种知名算法进行比较;实验结果表明ZLS-SMPSO-MM在决策空间能够找到更多的等价解,且整体性能要好于所比较算法。     

Population based Local Search for university course timetabling problems

Population based algorithms are generally better at exploring a search space than local search algorithms (i.e. searches based on a single heuristic). However, the limitation of many population based algorithms is in exploiting the search space. We propose a population based Local Search (PB-LS) heuristic that is embedded within a local search algorithm (as a mechanism to exploit the search space). PB-LS employs two operators. The first is applied to a single solution to determine the force between the incumbent solution and the trial current solution (i.e. a single direction force), whilst the second operator is applied to all solutions to determine the force in all directions. The progress of the search is governed by these forces, either in a single direction or in all directions. Our proposed algorithm is able to both diversify and intensify the search more effectively, when compared to other local search and population based algorithms. We use university course timetabling (Socha benchmark datasets) as a test domain. In order to evaluate the effectiveness of PB-LS, we perform a comparison between the performances of PB-LS with other approaches drawn from the scientific literature. Results demonstrate that PB-LS is able to produce statistically significantly higher quality solutions, outperforming many other approaches on the Socha dataset.     

Graphical model construction based on evolutionary algorithms

1 Introduction Evolutionary algorithms(EAs) [1～5] are stochastic search and optimization techniques, which were inspired by the analogy of evolution and population genetics. They have been demonstrated to be effective and robust in searching very large, varied, spaces in a wide range of applications, including classification, machine learning, ecological, so- cial systems and so on. However, most of the common evo- lutionary algorithms using simple operators are incapable of learning the reg…     

Graphical model construction based on evolutionary algorithms

Using Bayesian networks to model promising solutions from the current population of the evolutionary algorithms can ensure efficiency and intelligence search for the optimum. However, to construct a Bayesian network that fits a given dataset is a NP-hard problem, and it also needs consuming mass computational resources. This paper develops a methodology for constructing a graphical model based on Bayesian Dirichlet metric. Our approach is derived from a set of propositions and theorems by researching the local metric relationship of networks matching dataset. This paper presents the algorithm to construct a tree model from a set of potential solutions using above approach. This method is important not only for evolutionary algorithms based on graphical models, but also for machine learning and data mining. The experimental results show that the exact theoretical results and the approximations match very well.     

一种基于密度和网格的簇心可确定聚类算法

以网格化数据集来减少聚类过程中的计算复杂度,提出一种基于密度和网格的簇心可确定聚类算法.首先网格化数据集空间,以落在单位网格对象里的数据点数表示该网格对象的密度值,以该网格到更高密度网格对象的最近距离作为该网格的距离值;然后根据簇心网格对象同时拥有较高的密度和较大的距离值的特征,确定簇心网格对象,再通过一种基于密度的划分方式完成聚类;最后,在多个数据集上对所提出算法与一些现有聚类算法进行聚类准确性与执行时间的对比实验,验证了所提出算法具有较高的聚类准确性和较快的执行速度.     

Enhancing genetic algorithms for dependent job scheduling in grid computing environments

Genetic Algorithms (GAs) are stochastic search techniques based on principles of natural selection and recombination that attempt to find optimal solutions in polynomial time by manipulating a population of candidate solutions. GAs have been widely used for job scheduling optimisation in both homogeneous and heterogeneous computing environments. When compared with list scheduling heuristics, GAs can potentially provide better solutions but require much longer processing time and significant experimentation to determine GA parameters. This paper presents a GA for scheduling dependent jobs in grid computing environments. A?number of selection and pre-selection criteria for the GA are evaluated with an aim to improve GA performance in job scheduling optimization. A?Task Matching with Data scheme is proposed as a GA mutation operator. Furthermore, the effect of the choice of heuristics for seeding the GA is investigated.     

Optimization of heuristic search using recursive algorithm selection and reinforcement learning

The traditional approach to computational problem solving is to use one of the available algorithms to obtain solutions for all given instances of a problem. However, typically not all instances are the same, nor a single algorithm performs best on all instances. Our work investigates a more sophisticated approach to problem solving, called Recursive Algorithm Selection, whereby several algorithms for a problem (including some recursive ones) are available to an agent that makes an informed decision on which algorithm to select for handling each sub-instance of a problem at each recursive call made while solving an instance. Reinforcement learning methods are used for learning decision policies that optimize any given performance criterion (time, memory, or a combination thereof) from actual execution and profiling experience. This paper focuses on the well-known problem of state-space heuristic search and combines the A* and RBFS algorithms to yield a hybrid search algorithm, whose decision policy is learned using the Least-Squares Policy Iteration (LSPI) algorithm. Our benchmark problem domain involves shortest path finding problems in a real-world dataset encoding the entire street network of the District of Columbia (DC), USA. The derived hybrid algorithm exhibits better performance results than the individual algorithms in the majority of cases according to a variety of performance criteria balancing time and memory. It is noted that the proposed methodology is generic, can be applied to a variety of other problems, and requires no prior knowledge about the individual algorithms used or the properties of the underlying problem instances being solved.