Reinforcement Learning Based Algorithms for Average Cost Markov Decision Processes

This article proposes several two-timescale simulation-based actor-critic algorithms for solution of infinite horizon Markov Decision Processes with finite state-space under the average cost criterion. Two of the algorithms are for the compact (non-discrete) action setting while the rest are for finite-action spaces. On the slower timescale, all the algorithms perform a gradient search over corresponding policy spaces using two different Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates. On the faster timescale, the differential cost function corresponding to a given stationary policy is updated and an additional averaging is performed for enhanced performance. A proof of convergence to a locally optimal policy is presented. Next, we discuss a memory efficient implementation that uses a feature-based representation of the state-space and performs TD(0) learning along the faster timescale. The TD(0) algorithm does not follow an on-line sampling of states but is observed to do well on our setting. Numerical experiments on a problem of rate based flow control are presented using the proposed algorithms. We consider here the model of a single bottleneck node in the continuous time queueing framework. We show performance comparisons of our algorithms with the two-timescale actor-critic algorithms of Konda and Borkar (1999) and Bhatnagar and Kumar (2004). Our algorithms exhibit more than an order of magnitude better performance over those of Konda and Borkar (1999).

Super-Streaming: A New Object Delivery Paradigm for Continuous Media Servers

A number of studies have focused on the design of continuous media, CM, (e.g., video and audio) servers to support the real-time delivery of CM objects. These systems have been deployed in local environments such as hotels, hospitals and cruise ships to support media-on-demand applications. They typically stream CM objects to the clients with the objective of minimizing the buffer space required at the client site. This objective can now be relaxed due to the availability of inexpensive storage devices at the client side. Therefore, we propose a Super-streaming paradigm that can utilize the client side resources in order to improve the utilization of the CM server. To support super-streaming, we propose a technique to enable the CM servers to deliver CM objects at a rate higher than their display bandwidth requirement. We also propose alternative admission control policies to downgrade super-streams in favor of regular streams when the resources are scarce. We demonstrate the superiority of our paradigm over streaming with both analytical and simulation models.Moreover, new distributed applications such as distant-learning, digital libraries, and home entertainment require the delivery of CM objects to geographically disbursed clients. For quality purposes, recently many studies proposed dedicated distributed architectures to support these types of applications. We extend our super-streaming paradigm to be applicable in such distributed architectures. We propose a sophisticated resource management policy to support super-streaming in the presence of multiple servers, network links and clients. Due to the complexity involved in modeling these architectures, we only evaluate the performance of super-streaming by a simulation study.     

Fluorescence in situ hybridization of three oncogenes on a leiomyoma

Using fluorescence in situ hybridization technique, expression of three oncogenes, C-myc, RARa, and cyclin-D was tested on a uterine leiomyoma. C-myc and RARa were amplified in approximately 30% and 90% of the cells, respectively. Numerous small signals of C-myc were indicative of the presence of double minutes. Amplification of RARa is being reported for the first time in a leiomyoma. Cyclin-D was normal in diploid cells while it was highly amplified in polyploid cells. Low levels of amplified C-myc and cyclin-D cells seem to be the reason for this tumor to be benign, while RARa could not be effective without the association of some other gene such as PML. Information presented here are significant toward developing new curative strategies such as gene-specific drugs and molecular manipulation to stop the activity of cancer gene. Further study may elucidate that how fibroids grow and maintain their rare benign nature.     

The utilization of soft transformation and genetic algorithm to model two sources of uncertainty of Indicator Kriging

Indicator Kriging (IK) is a geostatistical method that uses observation points to quantify the probabilities at which a set of thresholds are exceeded at unmeasured points. To improve IK accuracy, the interpolation process should consider its uncertainty sources. By doing this, we also maintain its ability to provide the conditional cumulative distribution function (ccdf), which is a reliable measure of local uncertainty. This study modeled two IK uncertainty sources, i.e., measurement errors attached to observation points and subjective threshold choices. Soft Indicator Kriging (SIK), which uses a soft transformation for observation points, considers the measurement errors of these two sources. To select the thresholds objectively, a genetic algorithm (GA) was performed to obtain the optimum set of thresholds related to an objective function, which minimized the mean absolute error (MAE).     

Multi-product multi-chance-constraint stochastic inventory control problem with dynamic demand and partial back-ordering: A harmony search algorithm

In this paper, a multiproduct inventory control problem is considered in which the periods between two replenishments of the products are assumed independent random variables, and increasing and decreasing functions are assumed to model the dynamic demands of each product. Furthermore, the quantities of the orders are assumed integer-type, space and budget are constraints, the service-level is a chance-constraint, and that the partial back-ordering policy is taken into account for the shortages. The costs of the problem are holding, purchasing, and shortage. We show the model of this problem is an integer nonlinear programming type and to solve it, a harmony search approach is used. At the end, three numerical examples of different sizes are given to demonstrate the applicability of the proposed methodology in real world inventory control problems, to validate the results obtained, and to compare its performances with the ones of both a genetic and a particle swarm optimization algorithms.     

An optimal procedure for minimizing total weighted resource tardiness penalty costs in the resource-constrained project scheduling problem

We present an optimal solution procedure for minimizing total weighted resource tardiness penalty costs in the resource-constrained project scheduling problem. In this problem, we assume the constrained renewable resources are limited to very expensive equipments and machines that are used in other projects and are not available in all periods of time of a project. In other words, for each resource, there is a dictated ready date as well as a due date such that no resource can be available before its ready date but the resources are permitted to be used after their due dates by paying penalty cost depending on the resource type. We also assume that only one unit of each resource type is available and no activity needs more than it for execution. The objective is to determine a schedule with minimal total weighted resource tardiness penalty costs. For this purpose, we present a branch-and-bound algorithm in which the branching scheme starts from a graph representing a set of conjunctions (the classical finish-start precedence constraints) and disjunctions (introduced by the resource constraints). In the search tree, each node is branched to two child nodes based on the two opposite directions of each undirected arc of disjunctions. Selection sequence of undirected arcs in the search tree affects the performance of the algorithm. Hence, we developed different rules for this issue and compare the performance of the algorithm under these rules using a randomly generated benchmark problem set.     

A new unsupervised feature selection method for text clustering based on genetic algorithms

Nowadays a vast amount of textual information is collected and stored in various databases around the world, including the Internet as the largest database of all. This rapidly increasing growth of published text means that even the most avid reader cannot hope to keep up with all the reading in a field and consequently the nuggets of insight or new knowledge are at risk of languishing undiscovered in the literature. Text mining offers a solution to this problem by replacing or supplementing the human reader with automatic systems undeterred by the text explosion. It involves analyzing a large collection of documents to discover previously unknown information. Text clustering is one of the most important areas in text mining, which includes text preprocessing, dimension reduction by selecting some terms (features) and finally clustering using selected terms. Feature selection appears to be the most important step in the process. Conventional unsupervised feature selection methods define a measure of the discriminating power of terms to select proper terms from corpus. However up to now the valuation of terms in groups has not been investigated in reported works. In this paper a new and robust unsupervised feature selection approach is proposed that evaluates terms in groups. In addition a new Modified Term Variance measuring method is proposed for evaluating groups of terms. Furthermore a genetic based algorithm is designed and implemented for finding the most valuable groups of terms based on the new measure. These terms then will be utilized to generate the final feature vector for the clustering process . In order to evaluate and justify our approach the proposed method and also a conventional term variance method are implemented and tested using corpus collection Reuters-21578. For a more accurate comparison, methods have been tested on three corpuses and for each corpus clustering task has been done ten times and results are averaged. Results of comparing these two methods are very promising and show that our method produces better average accuracy and F1-measure than the conventional term variance method.     

A general computational recognition primed decision model with multi-agent rescue simulation benchmark

Analytical decision making strategies rely on weighing pros and cons of multiple options in an unbounded rationality manner. Contrary to these strategies, recognition primed decision (RPD) model which is a primary naturalistic decision making (NDM) approach assumes that experienced and professional decision makers when encounter problems in real operating conditions are able to use their previous experiences and trainings in order to diagnose the problem, recall the appropriate solution, evaluate it mentally, and implement it to handle the problem in a satisficing manner. In this paper, a computational form of RPD, now called C-RPD, is presented. Unified Modeling Language was used as a modeling language to represent the proposed C-RPD model in order to make the implementation easy and obvious. To execute the model, RoboCup Rescue agent simulation environment, which is one of the best and the most famous complex and multi-agent large-scale environments, was selected. The environment simulates the incidence of fire and earthquakes in urban areas where it is the duty of the police forces, firefighters and ambulance teams to control the crisis. Firefighters of SOS team are first modeled and implemented by utilizing C-RPD and then the system is trained using an expert’s experience. There are two evaluations. To find out the convergence of different versions developed during experience adding, some of the developed versions are chosen and evaluated on seven maps. Results show performance improvements. The SOS team ranked first in an official world championship and three official open tournaments.     

Laser micro-welding of aluminum and copper with and without tin foil alloy

In this paper continuous laser welding of two dissimilar materials, aluminum and copper, was investigated. The aluminum and the copper utilized were Al3003-H14 and Cu110-H00, respectively. Two different sets of samples were laser welded; one in which a filler material, tin foil alloy (S-bond 220), was sandwiched between the aluminum and the copper and another set in which the aluminum and copper were directly welded without any filler. The foil alloy was utilized to enhance the compatibility of the two metals; aluminum and copper, reducing the brittleness of the intermetallic compound that may form and, subsequently, enhance the mechanical properties. The welding was carried out using an IPG 500 SM fiber laser. The length of the laser joint produced was 20 mm and the width was about 200 μm. The strength of the joint was evaluated by conducting the lap shear stress test. Samples in which filler foil was used exhibited a better performance in the lap shear stress test (an average of 780 N) than the samples without tin foil (an average of 650 N). The improvement in the lap shear test could be attributed to the positive effects of the filler on enhancing the compatibility of the intermetallic compound formed via diffusion. The fracture surface of both types of joints (with and without filler) was characterized using scanning electron microscope equipped with energy-dispersive X-ray (EDAX). To understand the failure initiation and propagation of the samples under tension, a finite element (FE) model was developed for the samples created with no filler material. The failure mechanism predicted from the FE model matches reasonably well with the experimental observations from EDAX analysis.