Variance-penalized Markov decision processes: dynamic programming and reinforcement learning techniques

In control systems theory, the Markov decision process (MDP) is a widely used optimization model involving selection of the optimal action in each state visited by a discrete-event system driven by Markov chains. The classical MDP model is suitable for an agent/decision-maker interested in maximizing expected revenues, but does not account for minimizing variability in the revenues. An MDP model in which the agent can maximize the revenues while simultaneously controlling the variance in the revenues is proposed. This work is rooted in machine learning/neural network concepts, where updating is based on system feedback and step sizes. First, a Bellman equation for the problem is proposed. Thereafter, convergent dynamic programming and reinforcement learning techniques for solving the MDP are provided along with encouraging numerical results on a small MDP and a preventive maintenance problem.     

Analysis of Clinic Layouts and Patient-Centered Procedural Innovations Using Discrete-Event Simulation

Abstract

Given that healthcare costs in the U.S. continue to climb, engineering managers can help address this situation by identifying designs for cost savings. Using discrete-event simulation, the layout of four different healthcare clinic scenarios considered in a redesign program named Patient Aligned Care Teams were studied. Model 1 is an existing layout in many clinics and Model 2 is staff-centered, while Models 3 and 4 provide additional space including common hallway(s). Our main conclusion is that Model 4 reduces patient travel distances and average queueing waits, while also providing the best privacy in accordance with HIPAA and the most security for the staff.     

Conjugation of curcumin with PVP capped gold nanoparticles for improving bioavailability

Curcumin, a natural polyphenolic compound, has astounding therapeutic applications but lacks in bioavailability mainly due to its poor solubility in water. Polyvinyl pyrrolidone (PVP) which is a proven drug carrier has been used to facilitate the conjugation of curcumin with gold nanoparticles and to improve the solubility of curcumin in water. In this conjugate diaryl heptanoid chromophore group of curcumin which is a much needed group in biomedical applications remains intact as observed from FTIR and UV–vis spectroscopy analysis. The work shows good promise for such conjugates as therapeutic-cum-imaging materials in biomedical field.     

Role of argon in hot wire chemical vapor deposition of hydrogenated nanocrystalline silicon thin films

Structural, optical and electrical properties of hydrogenated nanocrystalline silicon (nc-Si:H) films, deposited from silane (SiH₄) and argon (Ar) gas mixture without hydrogen by hot wire chemical vapor deposition (HW-CVD) method were investigated. Film properties are carefully and systematically studied as a function of argon dilution of silane (R_Ar). We observed that the deposition rate is much higher (4-23 Å/s) compared to conventional plasma enhanced chemical vapor deposited nc-Si:H films using Ar dilution of silane (0.5-0.83 Å/s). Characterization of these films with Raman spectroscopy revealed that Ar dilution of silane in HW-CVD endorses the growth of crystallinity and structural order in the nc-Si:H films. The Fourier transform infrared spectroscopic analysis showed that with increasing Ar dilution, the hydrogen bonding in the films shifts from di-hydrogen (Si-H₂) and (Si-H₂)_n complexes to mono-hydrogen (Si-H) bounded species. The hydrogen content in the films increases with increasing Ar dilution and was found to be < 4 at.% over the entire range of Ar dilutions of silane studied. However, the band gap shows decreasing trend with increase in Ar dilution of silane and it has been attributed to the decrease in the percentage of the amorphous phase in the film. The microstructure parameter was found to be > 0.4 for the films deposited at low Ar dilution of silane and ~ 0.1 or even less for the films deposited at higher Ar dilution, suggesting that there is an enhancement of structural order and homogeneity in the film. From the present study it has been concluded that the Ar dilution of silane is a key process parameter to induce the crystallinity and to improve the structural ordering in the nc-Si:H films deposited by the HW-CVD method.     

Simulation optimization for revenue management of airlines with cancellations and overbooking

This paper develops a model-free simulation-based optimization model to solve a seat-allocation problem arising in airlines. The model is designed to accommodate a number of realistic assumptions for real-world airline systems—in particular, allowing cancellations of tickets by passengers and overbooking of planes by carriers. The simulation–optimization model developed here can be used to solve both single-leg problems and multi-leg or network problems. A model-free simulation–optimization approach only requires a discrete-event simulator of the system along with a numerical optimization method such as a gradient-ascent technique or a meta-heuristic. In this sense, it is relatively “easy” because alternative models such as dynamic programming or model-based gradient-ascent usually require more mathematically involved frameworks. Also, existing simulation-based approaches in the literature, unlike the one presented here, fail to capture the dynamics of cancellations and overbooking in their models. Empirical tests conducted with our approach demonstrate that it can produce robust solutions which provide revenue improvements over heuristics used in the industry, namely, EMSR (Expected Marginal Seat Revenue) for single-leg problems and DAVN (Displacement Adjusted Virtual Nesting) for networks.     

Phyto-inspired silica nanowires: characterization and application in lipase immobilization

Silica nanostructures were phyto-fabricated on different surfaces by using pomegranate (Punica granatum) leaf extracts. On zinc films, nanowires were obtained. On other surfaces such as silica, alumina, zinc oxide, and glass, spherical aggregates, cubic assemblies, microflakes, and acicular structures, respectively, were observed. The nanowires developed on Zn surfaces were characterized by scanning electron microscope-energy dispersive spectrometer (SEM-EDS), X-ray diffraction (XRD), photoluminescence, and Fourier transform infrared (FTIR) spectroscopic analysis. XRD profiles displayed peaks at 2.4, 4.9, and 12.1° indicating the presence of silica nanostructures. When excited at 340 nm, the reaction mixtures displayed a characteristic blue luminescence at 404 nm. FTIR spectra showed the existence of Si-OH and Si-O-Si bonds. The nanowires were functionalized with amine groups and used for the covalent immobilization of Candida rugosa lipase. The immobilized enzyme displayed better pH and temperature stability and retained 80% activity after 20 cycles. This paper highlights a novel route for the phyto-mediated growth of silica nanowires on Zn surfaces, their characterization and effective use as a matrix for enzyme immobilization.     

An algorithm for sub-optimal routeing in series-parallel queueing networks

The optimal routeing problem of maximizing system throughput in series-parallel networks with finite buffers is studied in this paper. The problem is extremely difficult to solve since closed form expressions are not easily constructed for throughput in finite networks. A piece-wise linear upper bound on the throughput of a tandem network is used to develop a throughput approximation in seriesparallel networks. Based on this approximation we are able to specify a suboptimal range for routeing probabilities at each junction in the network as a function of the arrival rate to this junction. We also specify a unique value for the routeing probability at each junction, independent of the arrival rate to that junction. We then construct an O(N) algorithm to analyse general series-parallel networks with more than one junction and specify the sub-optimal routeing probabilities at each junction.