Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-Part I: I.I.D. rewards

At each instant of time we are required to sample a fixed numberm geq 1out ofNi.i.d, processes whose distributions belong to a family suitably parameterized by a real numbertheta. The objective is to maximize the long run total expected value of the samples. Following Lai and Robbins, the learning loss of a sampling scheme corresponding to a configuration of parametersC = (theta_{1},..., theta_{N})is quantified by the regretR_{n}(C). This is the difference between the maximum expected reward at timenthat could be achieved ifCwere known and the expected reward actually obtained by the sampling scheme. We provide a lower bound for the regret associated with any uniformly good scheme, and construct a scheme which attains the lower bound for every configurationC. The lower bound is given explicitly in terms of the Kullback-Liebler number between pairs of distributions. Part II of this paper considers the same problem when the reward processes are Markovian.     

Extensions of the multiarmed bandit problem: The discounted case

There areNindependent machines. Machine i is described by a sequence{X^{i}(s), F^{i}(s)}whereX^{i}(s)is the immediate reward and F^{i}(s) is the information available before i is operated for the sth lime. At each time one operates exacfiy one machine; idle machines remain frozen. The problem is to schedule the operation of the machines so as to maximize the expected total discounted sequence of rewards. An elementary proof shows that to each machine is associated an index, and the optimal policy operates the machine with the largest current index. When the machines are completely observed Markov chains, this coincides with the well-known Gittins index rule, and new algorithms are given for calculating the index. A reformulation of the bandit problem yields the tax problem, which includes, as a special case, Klimov's waiting time problem. Using the concept of superprocess, an index rule is derived for the case where new machines arrive randomly. Finally, continuous time versions of these problems are considered for both preemptive and nonpreemptive disciplines.     

A lemma on the multiarmed bandit problem

We prove a lemma on the optimal value function for the multiarmed bandit problem which provides a simple direct proof of optimality of writeoff policies. This, in turn, leads to a new proof of optimality of the index rule.     

An asymptotically optimal policy for finite support models in the multiarmed bandit problem

In the multiarmed bandit problem the dilemma between exploration and exploitation in reinforcement learning is expressed as a model of a gambler playing a slot machine with multiple arms. A policy chooses an arm in each round so as to minimize the number of times that arms with suboptimal expected rewards are pulled. We propose the minimum empirical divergence (MED) policy and derive an upper bound on the finite-time regret which meets the asymptotic bound for the case of finite support models. In a setting similar to ours, Burnetas and Katehakis have already proposed an asymptotically optimal policy. However, we do not assume any knowledge of the support except for its upper and lower bounds. Furthermore, the criterion for choosing an arm, minimum empirical divergence, can be computed easily by a convex optimization technique. We confirm by simulations that the MED policy demonstrates good performance in finite time in comparison to other currently popular policies.     

Optimal stopping problems for multiarmed bandit processes with arms' independence

This paper deals with the optimal stopping problem for multiarmed bandit processes. Under the assumption of independence of arms we show that optimal strategies and stopping times are expressed by the dynamic allocation indices for each arm. This paper reduces this problem to several independent one-parameter optimal stopping problems. On the basis of these results, we characterize optimal strategies and stopping times. Moreover, this paper also extends those to the case allowing time constraints. In the case where arm's state evolve according to Markov chains with finite state, linear programming calculation of optimal strategies and stopping times is discussed.     

Asymptotically efficient adaptive allocation schemes for controlledMarkov chains: finite parameter space

The authors consider a controlled Markov chain whose transition probabilities and initial distribution are parametrized by an unknown parameter &thetas; belonging to some known parameter space Θ. There is a one-step reward associated with each pair of control and the following state of the process. The objective is to maximize the expected value of the sum of one-step rewards over an infinite horizon. The loss associated with a control scheme at a parameter value is the function of time giving the difference between the maximum reward that could have been achieved if the parameter were known and the reward achieved by the scheme. Since it is impossible to minimize the loss uniformly for all parameter values, the authors define uniformly good adaptive control schemes and restrict attention to these schemes. They develop a lower bound on the loss associated with any uniformly good control scheme. They construct an adaptive control scheme whose loss equals the lower bound for every parameter value and is therefore asymptotically efficient     

An efficient memory-based electromagnetism-like mechanism for the redundancy allocation problem

Meta-heuristic algorithms have been successfully applied to solve the redundancy allocation problem in recent years. Among these algorithms, the electromagnetism-like mechanism (EM) is a powerful population-based algorithm designed for continuous decision spaces. This paper presents an efficient memory-based electromagnetism-like mechanism called MBEM to solve the redundancy allocation problem. The proposed algorithm employs a memory matrix in local search to save the features of good solutions and feed it back to the algorithm. This would make the search process more efficient. To verify the good performance of MBEM, various test problems, especially the 33 well-known benchmark instances in the literature, are examined. The experimental results show that not only optimal solutions of all benchmark instances are obtained within a reasonable computer execution time, but also MBEM outperforms EM in terms of the quality of the solutions obtained, even for large-size problems.     

Finite-time lower bounds for the two-armed bandit problem

We obtain minimax lower bounds on the regret for the classical two-armed bandit problem. We provide a finite-sample minimax version of the well-known log n asymptotic lower bound of Lai and Robbins (1985). The finite-time lower bound allows us to derive conditions for the amount of time necessary to make any significant gain over a random guessing strategy. These bounds depend on the class of possible distributions of the rewards associated with the arms. For example, in contrast to the log n asymptotic results on the regret, we show that the minimax regret is achieved by mere random guessing under fairly mild conditions on the set of allowable configurations of the two arms. That is, we show that for every allocation rule and for every n, there is a configuration such that the regret at time n is at least 1-ϵ times the regret of random guessing, where ϵ is any small positive constant     

Asymptotically efficient adaptive allocation schemes for controlledi.i.d. processes: finite parameter space

The authors consider a controlled i.i.d. (independently identically distributed) process whose distribution is parametrized by an unknown parameter &thetas; belonging to some known parameter space Θ, and a one-step reward associated with each pair of control and the following state of the process. The objective is to maximize the expected value of the sum of one-step rewards over an infinite horizon. By introducing the loss associated with a control scheme, it is shown that the problem is equivalent to minimizing this loss. Uniformly good adaptive control schemes are defined and emphasized. A lower bound on the loss associated with any uniformly good control scheme is developed. Finally, an adaptive control scheme is constructed whose loss equals the lower bound, and is therefore asymptotically efficient     

Benders decomposition for the uncapacitated multiple allocation hub location problem

In telecommunication and transportation systems, the uncapacitated multiple allocation hub location problem (UMAHLP) arises when we must flow commodities or information between several origin–destination pairs. Instead of establishing a direct node to node connection from an origin to its destination, the flows are concentrated with others at facilities called hubs. These flows are transported on links established between hubs, being then splitted and delivered to its final destination. Systems with this sort of topology are named hub-and-spoke (HS) systems or hub-and-spoke networks. They are designed to exploit the scale economies attainable through the shared use of high capacity links between hubs. Therefore, the problem is to find the least expensive HS network, selecting hubs and assigning traffic to them, given the demands between each origin–destination pair and the respective transportation costs. In the present paper, we present efficient Benders decomposition algorithms based on a well known formulation to tackle the UMAHLP. We have been able to solve some large instances, considered ‘out of reach’ of other exact methods in reasonable time.     

HubLocator: an exact solution method for the multiple allocation hub location problem

HubLocator is a new branch-and-bound procedure for the uncapacitated multiple allocation hub location problem. An existing optimal method developed by Klincewicz (Location Sci. 4 (1996) 173) is based on dual ascent and dual adjustment techniques applied to a disaggregated model formulation. These techniques have already successfully been used to solve the closely related simple plant location problem. However, due to the specific structure of the problem at hand, the success of these techniques in reducing the computational effort is rather restricted. Therefore, HubLocator additionally considers an aggregated model formulation enabling us to significantly tighten the lower bounds. Upper bounds which satisfy complementary slackness conditions for some constraints are constructed and improved by means of a simple heuristic procedure. Computational experiments demonstrate that optimal solutions for problems with up to 40 nodes can be found in a reasonable amount of time.Scope and purposeGround and air transportation networks, postal delivery networks, and computer networks are often configured as hub-and-spoke systems. Traffic between two locations is not transported directly between these locations, but routed via particular switching or consolidation points called hubs. Due to increased traffic on linkages between hubs, larger vehicles can be used or the capacity of existing vehicles can be utilized more efficiently, resulting in smaller per unit transportation costs. The exploitation of scale economies as a result of the reduced number of linkages, which have to be operated in a hub-and-spoke system, compared to a fully interconnected network is an important advantage of this type of system.Designing hub-and-spoke networks deals with the selection of hubs from a given set of potential locations and the routing of traffic. We consider a special type of such a hub location problem and adapt a successful technique developed to find an optimal solution for the well-known simple plant location problem.     

An efficient memetic algorithm for the uncapacitated single allocation hub location problem

In this paper, we present a memetic algorithm (MA) for solving the uncapacitated single allocation hub location problem (USAHLP). Two efficient local search heuristics are designed and implemented in the frame of an evolutionary algorithm in order to improve both the location and allocation part of the problem. Computational experiments, conducted on standard CAB/AP hub data sets (Beasley in J Global Optim 8:429–433, 1996) and modified AP data set with reduced fixed costs (Silva and Cunha in Computer Oper Res 36:3152–3165, 2009), show that the MA approach is superior over existing heuristic approaches for the USAHLP. For several large-scale AP instances up to 200 nodes, the MA improved the best-known solutions from the literature until now. Numerical results on instances with 300 and 400 nodes introduced in Silva and Cunha (Computer Oper Res 36:3152–3165, 2009) show significant improvements in the sense of both solution quality and CPU time. The robustness of the MA was additionally tested on a challenging set of newly generated large-scale instances with 520–900 nodes. To the best of our knowledge, these are the largest USAHLP problem dimensions solved in the literature until now. In addition, in this paper, we report for the first time optimal solutions for 30 AP and modified AP instances.     

An electromagnetism-like metaheuristic for the uncapacitated multiple allocation p-hub median problem

This paper deals with the uncapacitated multiple allocation p-hub median problem (UMApHMP). An electromagnetism-like (EM) method is proposed for solving this NP-hard problem. Our new scaling technique, combined with the movement based on the attraction–repulsion mechanism, directs the EM towards promising search regions. Numerical results on a battery of benchmark instances known from the literature are reported. They show that the EM reaches all previously known optimal solutions, and gives excellent results on large-scale instances. The present approach is also extended to solve the capacitated version of the problem. As it was the case in the uncapacitated version, EM also reached all previously known optimal solutions.     

New simple and efficient heuristics for the uncapacitated single allocation hub location problem

Hub-and-spoke networks are widely studied in the area of location theory. They arise in several contexts, including passenger airlines, postal and parcel delivery, and computer and telecommunication networks. Hub location problems usually involve three simultaneous decisions to be made: the optimal number of hub nodes, their locations and the allocation of the non-hub nodes to the hubs. In the uncapacitated single allocation hub location problem (USAHLP) hub nodes have no capacity constraints and non-hub nodes must be assigned to only one hub. In this paper, we propose three variants of a simple and efficient multi-start tabu search heuristic as well as a two-stage integrated tabu search heuristic to solve this problem. With multi-start heuristics, several different initial solutions are constructed and then improved by tabu search, while in the two-stage integrated heuristic tabu search is applied to improve both the locational and allocational part of the problem. Computational experiments using typical benchmark problems (Civil Aeronautics Board (CAB) and Australian Post (AP) data sets) as well as new and modified instances show that our approaches consistently return the optimal or best-known results in very short CPU times, thus allowing the possibility of efficiently solving larger instances of the USAHLP than those found in the literature. We also report the integer optimal solutions for all 80 CAB data set instances and the 12 AP instances up to 100 nodes, as well as for the corresponding new generated AP instances with reduced fixed costs.     

Path relinking approach for multiple allocation hub maximal covering problem

We consider the multiple allocation hub maximal covering problem (MAHMCP): Considering a serviced O–D flow was required to reach the destination optionally passing through one or two hubs in a limited time, cost or distance, what is the optimal way to locate p hubs to maximize the serviced flows? By designing a new model for the MAHMCP, we provide an evolutionary approach based on path relinking. The Computational experience of an AP data set was presented. And a special application on hub airports location of Chinese aerial freight flows between 82 cities in 2002 was introduced.     

Asymptotically efficient estimators for the fittings of coupled circles and ellipses

Fitting a pair of coupled geometric objects to a number of coordinate points is a challenging and important problem in many applications including coordinate metrology, petroleum engineering and image processing. This paper derives two asymptotically efficient estimators, one for concentric circles fitting and the other for concentric ellipses fitting, based on the weighted equation error formulation and non-linear parameter transformation. The Kanatani–Cramér–Rao (KCR) lower bounds for the parameter estimates of the concentric circles and concentric ellipses under zero-mean Gaussian noise are provided to serve as the performance benchmark. Small-noise analysis shows that the proposed estimators reach the KCR lower bound performance asymptotically. The accuracy of the proposed estimators is corroborated by experiments with synthetic data and realistic images.     

Matching statistics: efficient computation and a new practical algorithm for the multiple common substring problem

We present new algorithms for computing matching statistics with suffix arrays. We show how the Multiple Common Substring Problem can be solved efficiently in practice with a new approach using matching statistics. This problem consists of finding the common substrings of a set of strings. For the computation of matching statistics we compare seven different methods based on suffix trees and suffix arrays. Most of the suffix array algorithms have an inferior asymptotic worst case running time but a very low memory overhead and small constants in the running time complexity. Our experiments show a good performance in practice. Copyright © 2005 John Wiley & Sons, Ltd.     

The filtering problem for continuous-time linear systems with Markovian switching coefficients

The stochastic model under consideration is a Markovian jump process θ, with finite state space, feeding the parameters of a linear diffusion process x. The processes y and z observe linearly and separately x and θ in independent white noises. Some properties of the finite optimal filter for the x and θ processes given the history of measurements z are investigated. Apart from their theoretical interest, these results have an interesting practical bearing on the general filtering problem, by providing a natural finite suboptimal solution. Preliminary experimental results show the effectiveness of our approach to estimate the state trajectory, even with a relatively low signal-to-noise ratio on the measurement processes.     

CBAR: an efficient method for mining association rules

The discovery of association rules is an important data-mining task for which many algorithms have been proposed. However, the efficiency of these algorithms needs to be improved to handle real-world large datasets. In this paper, we present an efficient algorithm named cluster-based association rule (CBAR). The CBAR method is to create cluster tables by scanning the database once, and then clustering the transaction records to the k-th cluster table, where the length of a record is k. Moreover, the large itemsets are generated by contrasts with the partial cluster tables. This not only prunes considerable amounts of data reducing the time needed to perform data scans and requiring less contrast, but also ensures the correctness of the mined results. Experiments with the FoodMart transaction database provided by Microsoft SQL Server show that CBAR outperforms Apriori, a well-known and widely used association rule.