A global optimization method for semi-supervised clustering

In this paper, we adapt Tuy’s concave cutting plane method to the semi-supervised clustering. We also give properties of local optimal solutions of the semi-supervised clustering. Numerical examples show that this method can give a better solution than other semi-supervised clustering algorithms do.     

A stochastic approach to hotel revenue optimization

Owning to similar business nature, it should be possible to directly migrate successful airline revenue management techniques to the hotel domain. However, one of the salient differences between airlines and hotels is rarely highlighted—the network structure of length of stay or the displacement effect. The hotel patrons go from a first stay-over night to a last stay-over night in consecutive night stays. The arrival demands for multi-night stays and the lengths of stay are stochastic in nature.In this paper, we propose a network optimization model for hotel revenue management under an uncertain environment. The network optimization is in a stochastic programming formulation so as to capture the randomness of the unknown demand (unknown number of arrivals and length of stays). A novel approach of robust optimization techniques for stochastic programming is applied to solve the problem. We also discuss the strategies for hotel management to take into account of risk trade-off; different pricing policies; cancellations and no-show; early check-outs; extended stay and over-booking are discussed. We showed that our proposed model can be modified to adopt these strategic considerations.     

A hybrid approach to constrained global optimization

In this paper, we propose a novel hybrid global optimization method to solve constrained optimization problems. An exact penalty function is first applied to approximate the original constrained optimization problem by a sequence of optimization problems with bound constraints. To solve each of these box constrained optimization problems, two hybrid methods are introduced, where two different strategies are used to combine limited memory BFGS (L-BFGS) with Greedy Diffusion Search (GDS). The convergence issue of the two hybrid methods is addressed. To evaluate the effectiveness of the proposed algorithm, 18 box constrained and 4 general constrained problems from the literature are tested. Numerical results obtained show that our proposed hybrid algorithm is more effective in obtaining more accurate solutions than those compared to.     

A new real-coded stochastic Bayesian optimization algorithm for continuous global optimization

Estimation of distribution algorithms are considered to be a new class of evolutionary algorithms which are applied as an alternative to genetic algorithms. Such algorithms sample the new generation from a probabilistic model of promising solutions. The search space of the optimization problem is improved by such probabilistic models. In the Bayesian optimization algorithm (BOA), the set of promising solutions forms a Bayesian network and the new solutions are sampled from the built Bayesian network. This paper proposes a novel real-coded stochastic BOA for continuous global optimization by utilizing a stochastic Bayesian network. In the proposed algorithm, the new Bayesian network takes advantage of using a stochastic structure (that there is a probability distribution function for each edge in the network) and the new generation is sampled from the stochastic structure. In order to generate a new solution, some new structure, and therefore a new Bayesian network is sampled from the current stochastic structure and the new solution will be produced from the sampled Bayesian network. Due to the stochastic structure used in the sampling phase, each sample can be generated based on a different structure. Therefore the different dependency structures can be preserved. Before the new generation is generated, the stochastic network’s probability distributions are updated according to the fitness evaluation of the current generation. The proposed method is able to take advantage of using different dependency structures through the sampling phase just by using one stochastic structure. The experimental results reported in this paper show that the proposed algorithm increases the quality of the solutions on the general optimization benchmark problems.     

A connectionist approach to rule refinement

A novel approach to rule refinement based upon connectionism is presented. This approach is capable of performing rule deletion, rule addition, changing rule quality, and modification of rule strengths. The fundamental algorithm is referred to as the Consistent-Shift algorithm. Its basis for identifying incorrect connections is that incorrect connections will often undergo larger inconsistent weight shift that correct ones during training with correct samples. By properly adjusting the detection threshold, incorrect connections would be uncovered, which can then be deleted or modified. Deletion of incorrect connections and addition of correct connections then translate into various forms of rule refinement just mentioned. The viability of this approach is demonstrated empirically.     

New global optimization algorithms for model-based clustering

The Expectation-Maximization (EM) algorithm is a very popular optimization tool for mixture problems and in particular for model-based clustering problems. However, while the algorithm is convenient to implement and numerically very stable, it only produces local solutions. Thus, it may not achieve the globally optimal solution in problems that have a large number of local optima. This paper introduces several new algorithms designed to produce global solutions in model-based clustering. The building blocks for these algorithms are methods from the operations research literature, namely the Cross-Entropy (CE) method and Model Reference Adaptive Search (MRAS). One problem with applying these methods directly is the efficient simulation of positive definite covariance matrices. We propose several new solutions to this problem. One solution is to apply the principles of Expectation-Maximization updating, which leads to two new algorithms, CE-EM and MRAS-EM. We also propose two additional algorithms, CE-CD and MRAS-CD, which rely on the Cholesky decomposition. We conduct numerical experiments of varying complexity to evaluate the effectiveness of the proposed algorithms in comparison to classical EM. We find that although a single run of the new algorithms is slower than a single run of EM, all have the potential for producing significantly better solutions. We also find that although repeat application of EM may achieve similar results, our algorithms provide automated, data-driven decision rules which may significantly reduce the burden of searching for the global optimum.     

A Bayesian framework for deformable pattern recognition withapplication to handwritten character recognition

Deformable models have recently been proposed for many pattern recognition applications due to their ability to handle large shape variations. These proposed approaches represent patterns or shapes as deformable models, which deform themselves to match with the input image, and subsequently feed the extracted information into a classifier. The three components-modeling, matching, and classification-are often treated as independent tasks. In this paper, we study how to integrate deformable models into a Bayesian framework as a unified approach for modeling, matching, and classifying shapes. Handwritten character recognition serves as a testbed for evaluating the approach. With the use of our system, recognition is invariant to affine transformation as well as other handwriting variations. In addition, no preprocessing or manual setting of hyperparameters (e.g., regularization parameter and character width) is required. Besides, issues on the incorporation of constraints on model flexibility, detection of subparts, and speed-up are investigated. Using a model set with only 23 prototypes without any discriminative training, we can achieve an accuracy of 94.7 percent with no rejection on a subset (11,791 images by 100 writers) of handwritten digits from the NIST SD-1 dataset     

A stochastic local search algorithm for constrained continuous global optimization

This paper presents a new stochastic local search algorithm known as feasible–infeasible search procedure (FISP) for constrained continuous global optimization. The proposed procedure uses metaheuristic strategies for combinatorial optimization as well as combined strategies for exploring continuous spaces, which are applied to an efficient process in increasingly refined neighborhoods of current points. We show effectiveness and efficiency of the proposed procedure on a standard set of 13 well‐known test problems. Furthermore, we compare the performance of FISP with SNOPT (sparse nonlinear optimizer) and with few successful existing stochastic algorithms on the same set of test problems.     

Newtonian clustering: An approach based on molecular dynamics and global optimization

Given a data set, a dynamical procedure is applied to the data points in order to shrink and separate, possibly overlapping clusters. Namely, Newton's equations of motion are employed to concentrate the data points around their cluster centers, using an attractive potential, constructed specially for this purpose. During this process, important information is gathered concerning the spread of each cluster. In succession this information is used to create an objective function that maps each cluster to a local maximum. Global optimization is then used to retrieve the positions of the maxima that correspond to the locations of the cluster centers. Further refinement is achieved by applying the EM-algorithm to a Gaussian mixture model whose construction and initialization is based on the acquired information. To assess the effectiveness of our method, we have conducted experiments on a plethora of benchmark data sets. In addition we have compared its performance against four clustering techniques that are well established in the literature.     

A New Subdivision Algorithm for the Bernstein Polynomial Approach to Global Optimization

In this paper,an improved algorithm is proposed for unconstrained global optimization to tackle non-convex nonlinear multivariate polynomial programming problems.The proposed algorithm is based on the Bernstein polynomial approach.Novel features of the proposed algorithm are that it uses a new rule for the selection of the subdivision point,modified rules for the selection of the subdivision direction,and a new acceleration device to avoid some unnecessary subdivisions.The performance of the proposed algorithm is numerically tested on a collection of 16 test problems.The results of the tests show the proposed algorithm to be superior to the existing Bernstein algorithm in terms of the chosen performance metrics.     

A connectionist approach to generating oblique decision trees

Neural networks and decision tree methods are two common approaches to pattern classification. While neural networks can achieve high predictive accuracy rates, the decision boundaries they form are highly nonlinear and generally difficult to comprehend. Decision trees, on the other hand, can be readily translated into a set of rules. In this paper, we present a novel algorithm for generating oblique decision trees that capitalizes on the strength of both approaches. Oblique decision trees classify the patterns by testing on linear combinations of the input attributes. As a result, an oblique decision tree is usually much smaller than the univariate tree generated for the same domain. Our algorithm consists of two components: connectionist and symbolic. A three-layer feedforward neural network is constructed and pruned, a decision tree is then built from the hidden unit activation values of the pruned network. An oblique decision tree is obtained by expressing the activation values using the original input attributes. We test our algorithm on a wide range of problems. The oblique decision trees generated by the algorithm preserve the high accuracy of the neural networks, while keeping the explicitness of decision trees. Moreover, they outperform univariate decision trees generated by the symbolic approach and oblique decision trees built by other approaches in accuracy and tree size.     

A two-phase connectionist approach to invariant picture interpretation

The paper presents an efficient two-phase approach to picture interpretation based on original connectionist techniques. During the first phase invariant representations of individual objects are obtained based on third-order image correlations and appropriate neural network classifiers are used to provide a probabilistic assignment of labels to objects. The second phase uses relationships between objects to reduce or eliminate ambiguity by means of a relaxation scheme based on stochastic learning automata. Both phases are particularly suited to parallel implementation. Simulation experiments revealed the effectiveness of our approach in solving several problems of small and medium sizes.     

Comparison of deterministic and stochastic approaches to global optimization

In this paper, we compare two different approaches to nonconvex global optimization. The first one is a deterministic spatial Branch‐and‐Bound algorithm, whereas the second approach is a Quasi Monte Carlo (QMC) variant of a stochastic multi level single linkage (MLSL) algorithm. Both algorithms apply to problems in a very general form and are not dependent on problem structure. The test suite we chose is fairly extensive in scope, in that it includes constrained and unconstrained problems, continuous and mixed‐integer problems. The conclusion of the tests is that in general the QMC variant of the MLSL algorithm is generally faster, although in some instances the Branch‐and‐Bound algorithm outperforms it.     

Non-iterative approach for global mesh optimization

This paper presents a global optimization operator for arbitrary meshes. The global optimization operator is composed of two main terms, one part is the global Laplacian operator of the mesh which keeps the fairness and another is the constraint condition which reserves the fidelity to the mesh. The global optimization operator is formulated as a quadratic optimization problem, which is easily solved by solving a sparse linear system. Our global mesh optimization approach can be effectively used in at least three applications: smoothing the noisy mesh, improving the simplified mesh, and geometric modeling with subdivision-connectivity. Many experimental results are presented to show the applicability and flexibility of the approach.     

A fuzzy connectionist expert system for visual pattern classification

In this paper, a connectionist model to integrate knowledge-based techniques into neural network approaches for visual pattern classification is presented. We propose a new structure of connectionist model which has rule-following capability as well as instance-based learning capability. Each node of the proposed network is doubly linked by two types of connections: positive connection and negative connection. Such connectionism provides a methodology to construct the classifier from the rule base and allows the expert knowledge to be utilized for the effective learning. For visual pattern classification, we present the techniques for knowledge representation and utilization using the concepts of fuzzy rules and fuzzy relations. We also discuss in this paper some advantageous characteristics of the model: result explanation capability and rule refinement capability. From the experimental results of the handwritten digit classification, the feasibility of the proposed model is evaluated.     

Improved cat swarm optimization algorithm for solving global optimization problems and its application to clustering

This paper presents a cat swarm optimization (CSO) algorithm for solving global optimization problems. In CSO algorithm, some modifications are incorporated to improve its performance and balance between global and local search. In tracing mode of the CSO algorithm, a new search equation is proposed to guide the search toward a global optimal solution. A local search method is incorporated to improve the quality of solution and overcome the local optima problem. The proposed algorithm is named as Improved CSO (ICSO) and the performance of the ICSO algorithm is tested on twelve benchmark test functions. These test functions are widely used to evaluate the performance of new optimization algorithms. The experimental results confirm that the proposed algorithm gives better results than the other algorithms. In addition, the proposed ICSO algorithm is also applied for solving the clustering problems. The performance of the ICSO algorithm is evaluated on five datasets taken from the UCI repository. The simulation results show that ICSO-based clustering algorithm gives better performance than other existing clustering algorithms.     

A robust stochastic genetic algorithm (StGA) for global numerical optimization

Many real-life problems can be formulated as numerical optimization of certain objective functions. However, often an objective function possesses numerous local optima, which could trap an algorithm from moving toward the desired global solution. Evolutionary algorithms (EAs) have emerged to enable global optimization; however, at the present stage, EAs are basically limited to solving small-scale problems due to the constraint of computational efficiency. To improve the search efficiency, this paper presents a stochastic genetic algorithm (StGA). A novel stochastic coding strategy is employed so that the search space is dynamically divided into regions using a stochastic method and explored region-by-region. In each region, a number of children are produced through random sampling, and the best child is chosen to represent the region. The variance values are decreased if at least one of five generated children results in improved fitness, otherwise, the variance values are increased. Experiments on 20 test functions of diverse complexities show that the StGA is able to find the near-optimal solution in all cases. Compared with several other algorithms, StGA achieves not only an improved accuracy, but also a considerable reduction of the computational effort. On average, the computational cost required by StGA is about one order less than the other algorithms. The StGA is also shown to be able to solve large-scale problems.     

Rotation-invariant neural pattern recognition system withapplication to coin recognition

In pattern recognition, it is often necessary to deal with problems to classify a transformed pattern. A neural pattern recognition system which is insensitive to rotation of input pattern by various degrees is proposed. The system consists of a fixed invariance network with many slabs and a trainable multilayered network. The system was used in a rotation-invariant coin recognition problem to distinguish between a 500 yen coin and a 500 won coin. The results show that the approach works well for variable rotation pattern recognition.     

Social group optimization for global optimization of multimodal functions and data clustering problems

Neural Computing and Applications - Cost and physical constraints in the engineering applied problems obligate finding the best results that global optimization algorithms cannot realize. For...