Reinforcement Learning Based Algorithms for Average Cost Markov Decision Processes

This article proposes several two-timescale simulation-based actor-critic algorithms for solution of infinite horizon Markov Decision Processes with finite state-space under the average cost criterion. Two of the algorithms are for the compact (non-discrete) action setting while the rest are for finite-action spaces. On the slower timescale, all the algorithms perform a gradient search over corresponding policy spaces using two different Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates. On the faster timescale, the differential cost function corresponding to a given stationary policy is updated and an additional averaging is performed for enhanced performance. A proof of convergence to a locally optimal policy is presented. Next, we discuss a memory efficient implementation that uses a feature-based representation of the state-space and performs TD(0) learning along the faster timescale. The TD(0) algorithm does not follow an on-line sampling of states but is observed to do well on our setting. Numerical experiments on a problem of rate based flow control are presented using the proposed algorithms. We consider here the model of a single bottleneck node in the continuous time queueing framework. We show performance comparisons of our algorithms with the two-timescale actor-critic algorithms of Konda and Borkar (1999) and Bhatnagar and Kumar (2004). Our algorithms exhibit more than an order of magnitude better performance over those of Konda and Borkar (1999).

Four degree of freedom liquid dispenser for direct write capillary self-assembly with sub-nanoliter precision

Capillary forces provide a ubiquitous means of organizing micro- and nanoscale structures on substrates. In order to investigate the mechanism of capillary self-assembly and to fabricate complex ordered structures, precise control of the meniscus shape is needed. We present a precision instrument that enables deposition of liquid droplets spanning from 2 nl to 300 μl, in concert with mechanical manipulation of the liquid-substrate interface with four degrees of freedom. The substrate has sub-100 nm positioning resolution in three axes of translation, and its temperature is controlled using thermoelectric modules. The capillary tip can rotate about the vertical axis while simultaneously dispensing liquid onto the substrate. Liquid is displaced using a custom bidirectional diaphragm pump, in which an elastic membrane is hydraulically actuated by a stainless steel syringe. The syringe is driven by a piezoelectric actuator, enabling nanoliter volume and rate control. A quantitative model of the liquid dispenser is verified experimentally, and suggests that compressibility in the hydraulic line deamplifies the syringe stroke, enabling sub-nanoliter resolution control of liquid displacement at the capillary tip. We use this system to contact-print water and oil droplets by mechanical manipulation of a liquid bridge between the capillary and the substrate. Finally, we study the effect of droplet volume and substrate temperature on the evaporative self-assembly of monodisperse polymer microspheres from sessile droplets, and demonstrate the formation of 3D chiral assemblies of micro-rods by rotation of the capillary tip during evaporative assembly.     

A simulation‐based algorithm for optimal pricing policy under demand uncertainty

We propose a simulation‐based algorithm for computing the optimal pricing policy for a product under uncertain demand dynamics. We consider a parameterized stochastic differential equation (SDE) model for the uncertain demand dynamics of the product over the planning horizon. In particular, we consider a dynamic model that is an extension of the Bass model. The performance of our algorithm is compared to that of a myopic pricing policy and is shown to give better results. Two significant advantages with our algorithm are as follows: (a) it does not require information on the system model parameters if the SDE system state is known via either a simulation device or real data, and (b) as it works efficiently even for high‐dimensional parameters, it uses the efficient smoothed functional gradient estimator.     

A grey approach for forecasting in a supply chain during intermittentdisruptions

The supply chains today have become vulnerable to frequent disruptions, and with continuing emphasis on efficiency, lacks robustness to deal with them. A part of the solution lies in forecasting the disruption beforehand and the other part in knowing which policies will suit such disrupted conditions best. Accurate and immediate forecasts are a must in a supply chain and hence play a huge role in stabilizing. This study compares the performance of three established forecasting methods (moving average, weighted moving average and exponential smoothing) as well as grey prediction method, during disruptions and stable situations. The experiments are performed in the form of discrete event simulation, on a four stage beer game settings. The results show that moving average and weighted moving average methods become incompetent during disruptions, and are useful only during stable times, when the demand hovers around a predefined mean value. Exponential smoothing and grey method seems to give better results during disruptions and also during stable times in upstream tiers. Grey prediction method in particular is the best method when the disruption frequency is high and also when the disruption impact is gradual rather than sudden.     

A novel image fusion framework for night-vision navigation and surveillance

The night-vision image fusion plays a critical role in detecting targets and obstructions in low light or total darkness, which has great importance for pedestrian recognition, vehicle navigation, surveillance and monitoring applications. The central idea is to fuse low-light visible and infrared imagery into a single output. In this paper, we describe a new fusion framework for spatially registered visual and infrared images. The proposed framework utilizes the properties of fractal dimension and phase congruency in the non-subsampled contourlet transform (NSCT) domain. The proposed framework applies multiscale NSCT on visual and IR images to get low- and high-frequency bands. The varied frequency bands of the transformed images are then fused while exploiting their characteristics. Finally, the inverse NSCT is performed to get the fused image. The performance of the proposed framework is validated by extensive experiments on different scene imaginary, where the definite advantages are demonstrated subjectively and objectively.     

Effect of particle size of magnesium silicate filler on physical properties of paper

Fillers are essential component of printing papers to increase the opacity, brightness, and to improve formation and printing properties. As a very little work has been reported so far on magnesium silicate (talc), the study was conducted with the filler of different particle size for papermaking. The sheets were made in the laboratory with refined mixed hardwood chemical pulp with five grades of talc, ground calcium carbonate (GCC) and precipitated calcium carbonate (PCC) fillers with 15–24% ash level. Apparent density along with tensile, burst, tear index, Z‐direction tensile strength (ZDTS) and bending stiffness index were evaluated for talc filled sheets, and compared with GCC and PCC. Physical strength properties of talc filled sheets were decreased at a faster rate on increasing filler loading in paper and decreasing the particle size of the filler. With same type of filler its particle size determines the physical properties of paper. The postulate was not found to be valid for all the three varieties of fillers viz., talc, PCC and GCC. Shape and geometry of the PCC and GCC fillers determine the individual property. © 2012 Canadian Society for Chemical Engineering     

Stability of Cold-Pressed Oil from Commercial Indian Niger (Guizotia abyssinica (L.f.) Cass.) Seed as affected by Blending and Interesterification

Blending and interesterification of cold‐pressed oil from commercially available niger (Guizotia abyssinica (L.f.) Cass.) seeds was performed to improve its stability. The fatty acid composition of cold‐pressed niger seed oil (NSO) revealed that it contained a huge amount of polyunsaturated linoleic acid (69.2 %). NSO being rich in polyunsaturated fatty acids (PUFA) was susceptible to oxidation and hence was blended with saturated fatty acid (SFA) rich coconut oil (CNO) and monounsaturated fatty acid (MUFA) rich olive–pomace oil (OO) to enhance its stability. CNO contained a total of 91.3 % of SFA, while OO had oleic acid, C18:1 (74.3 %) as MUFA. Two blends of NSO with CNO and OO, i.e. NSO + CNO(B) and NSO + OO(B), were prepared in the ratio of 1:1. The blends were further interesterified using the lipase enzyme from Rhizomucor meihei and interesterified oils, i.e. NSO + CNO(I) and NSO + OO(I), were obtained. The oxidative stability of the oils was evaluated by incubating them at 37 °C and 55 % relative humidity (RH) for a period of 45 days. The peroxide values of NSO + CNO(B), NSO + OO(B), NSO + CNO(I) and NSO + OO(I) showed a reduction by 53.3, 42.6, 65.3 and 55.4 %, respectively, while the conjugated diene values showed a reduction by 75.0, 66.9, 76.7 and 75.3 %, respectively, as compared to NSO during the incubation period. This is probably the first report on the stability improvement of niger seed oil through blending and interesterification.