On the value of learning for Bernoulli bandits with unknown parameters

Investigates the multiarmed bandit problem, where each arm generates an infinite sequence of Bernoulli distributed rewards. The parameters of these Bernoulli distributions are unknown and initially assumed to be beta-distributed. Every time a bandit is selected, its beta-distribution is updated to new information in a Bayesian way. The objective is to maximize the long-term discounted rewards. We study the relationship between the necessity of acquiring additional information and the reward. This is done by considering two extreme situations, which occur when a bandit has been played N times: the situation where the decision maker stops learning and the situation where the decision maker acquires full information about that bandit. We show that the difference in reward between this lower and upper bound goes to zero as N grows large.     

Optimal resource allocation in synchronized multi-tier Internet services

Modern Internet systems have evolved from simple monolithic systems to complex multi-tiered architectures. For these systems, providing good response time performance is crucial for commercial success. In practice, the response-time performance of multi-tiered systems is often degraded by severe synchronization problems, causing jobs to wait for responses from different tiers. Synchronization between different tiers is a complicating factor in the optimal control and analysis of performance. In this paper, we study a generic multi-tier model with synchronization in a queuing-theoretical setting. The system is able to share processing capacity between arriving jobs that need to be sent to other tiers and the responses that have arrived after processing from these tiers. We provide structural results on the optimal resource allocation policy and provide a full characterization of the policy in the framework of Markov decision theory. We also highlight important effects of synchronization in the model. We validate our expressions through extensive experimentations for a wide range of resource configurations.     

A queueing model for call blending in call centers

Call centers that apply call blending obtain high-productivity and high-service levels by dynamically mixing inbound and outbound traffic. We show that agents should be assigned to outbound calls if the number of available agents exceeds a certain threshold. This control policy is optimal for equal service time distributions and a very good approximation otherwise.     

A simple index rule for efficient traffic splitting over parallel wireless networks with partial information

Multi-path communication solutions provide a promising means to improve the network performance in areas covered by multiple wireless access networks. Today, little is known about how to effectively exploit this potential. We study a model where flows are transferred over multiple parallel networks, each of which is modeled as a processor sharing node. The goal is to minimize the expected transfer time of elastic data traffic by smartly dispatching the flows to the networks, based on partial information about the numbers of foreground and background flows in each of the nodes. In the case of full state information, the optimal policy can be derived via standard MDP-techniques, but for models with partial information an optimal solution is hard to obtain. An important requirement is that the splitting algorithm is efficient, yet simple, easy-to-implement, scalable in the number of parallel networks and robust against changes in the parameter settings. We propose a simple index rule for splitting traffic streams based on partial information, and benchmark the results against the optimal solution in the case of full state information. Extensive simulations with real networks show that this method performs extremely well under practical circumstances for a wide range of realistic parameter settings.     

Optimal resource allocation for time-reservation systems

This paper studies the optimal resource allocation in time-reservation systems. Customers arrive at a service facility and receive service in two steps; in the first step information is gathered from the customer, which is then sent to a pool of computing resources, and in the second step the information is processed after which the customer leaves the system. A central decision maker has to decide when to reserve computing power from the pool of resources, such that the customer does not have to wait for the start of the second service step and that the processing capacity is not wasted due to the customer still being serviced at the first step. The decision maker simultaneously has to decide on how many processors to allocate for the second processing step such that reservation and holding costs are minimized. Since an exact analysis of the system is difficult, we decompose the system into two parts which are solved sequentially leading to nearly optimal solutions. We show via dynamic programming that the near-optimal number of processors follows a step function with as an extreme policy the bang-bang control. Moreover, we provide new fundamental insights in the dependence of the near-optimal policy on the distribution of the information gathering times. Numerical experiments demonstrate that the near-optimal policy closely matches the performance of the optimal policy of the original problem.