Utility,probabilistic constraints,mean and variance of discounted rewards in Markov decision processes

Summary This paper deals with discounted Markov decision processes, Markov with respect to a finite statespaceI, where for eachiI, and each decision epocht, there is a finite action space K(i, t). The paper is concerned with problems which are formulated in terms of the discounted rewards in several ways. In order to ensure that optimal, or near optimal, policies are obtained, the state spaceI is extended to augmented state-spaces A(n), or A(), including the accumulated discounted rewards. Specimen problems are formulated and some computational aspects examined.

---

Zusammenfassung Es werden diskontierte Markovsche Entscheidungsprozesse behandelt mit endlichem Zustandsraum I, wobei die Mengen der zulässigen Entscheidungen K(i, t) vom Zustandi I und vom Zeitpunktt abhängen können. Es werden verschiedene Zielfunktionen betrachtet, die jeweils als Funktion des diskontierten Gesamtgewinns (nicht dessen Erwartungswerts) formuliert werden. Um optimale oder fast-optimale Politiken zu erhalten ohne die gesamte Vorgeschichte zu registrieren, wird der Zustandsraum um die akkumulierten diskontierten Auszahlungen erweitert. Eine Auswahl solcher Probleme wird exemplarisch diskutiert einschließlich einiger Aspekte der numerischen Behandlung.

     

Reward revision and the average reward markov decision process

Summary We integrate two numerical procedures for solving the average reward Markov decision process (MDP), standard successive approximations and modified policy iteration with reward revision. Reward revision is the process of revising the reward structure of a second, more computationally desirable MDP so as to produce, in the limit, an optimality equation having a fixed point identical to that associated with the original MDP. A numerical study indicates that for MDP's having a non-sparse transition structure with a small number of relatively large entries per row, the addition of reward revision can have significant computational benefits.

---

Zusammenfassung Zur Lösung Markovscher Entscheidungsprozesse (MDP) mit Durchschnitts-Kriterium werden zwei numerische Verfahren, nämlich sukzessive Approximation und modifizierte Politik-Iteration, mit einer Transformation, der sogenannten 'Reward-Revision, kombiniert. Bei dieser Transformation werden die Übergangswahrscheinlichkeiten so abgeändert (ausgedünnt), daß das neue Modell sich numerisch günstiger verhält. Dazu müssen die einstufigen Erträge so revidiert werden, daß die Optimalitäts-Gleichung des neuen Modells im Limes mit der des ursprünglichen übereinstimmt. Numerische Untersuchungen zeigen, daß für MDP mit stark besetzten Übergangsmatrizen, bei denen nur an wenigen Stellen je Zeile große Werte stehen, die Anwendung von 'Reward Revision zu wesentlichen Einsparungen an Rechenaufwand führen kann.

---

---

Research supported by NSF Grant ECS-8319355     

Direct evidence for chain orientation in poly(acetylene)

Two conflicting structural models (platelet and fibril) have been proposed in the literature for pristine cis-poly(acetylene). The respective models have separate, distinct implications for the chemical, physical and electrical properties of the material, in particular, the cis-trans isomerization, doping and conduction mechanisms. Luttinger-type cis-poly(acetylene) was prepared and studied in the transmission electron microscope (TEM) by low-dose conventional imaging, in dark-field mode and by electron diffraction. Thin ‘films’ of the material were shown to consist of fibrillar nets, having fibrils of diameter 5–25 nm. Combined electron diffraction and dark-field observations in the TEM indicated the structural arrangement within the fibrils as that having polymer molecular chains aligned parallel to fibril axes.     

Frying Performance of No-<Emphasis Type="Italic">trans</Emphasis>, Low-Linolenic Acid Soybean Oils

Two extruded-expelled physically refined soybean oils with reduced contents of linolenic acid, ultra-low- linolenic acid (ULL, 1.5%) and low-linolenic acid (LL, 2.6%), and a extruded-expelled physically refined control oil (control, 5.3% linolenic acid) were evaluated by frying French fries in a commercial-like setting for 6 h day⁻¹ during 23 days. The oils became darker, increased in yellow color at the beginning, and became redder and less green throughout the process. Free fatty acids levels were not different among the oils until day 14, after which, ULL was different from the control for the remainder of frying. The conjugated dienoic acid values were greatest in the control. Generally, ULL and LL oils had lower percentages of polar compounds than did the control, providing a frying life 2 days longer than the control and ~30% increase in frying time. A trained sensory panel evaluated the French fries on days 2, 5, and 6. Buttery and potato flavors decreased, and rancid and painty flavors increased over frying time for all products. Rancid flavor was highest in the fries from the control oil. Overall, the ULL and LL oils performed better than did the control oil and ULL tended to perform better than the LL.     

Structure development in biaxially stretched polystyrene film: Part II. Theoretical analysis of orientation development

The present paper describes an analysis of the orientation developed in biaxially stretched polystyrene films in terms of the kinematics of the process and a viscoelastic constitutive equation. It was possible to achieve reasonable agreement of predicted and experimental birefringences through use of a convected Maxwell model and the Rheo-Optical Law.     

Diffusion on Some Simple Stratified Spaces

A variety of different imaging techniques produce data which naturally lie in stratified spaces. These spaces consist of smooth regions of maximal dimension glued together along lower dimensional boundaries. Diffusion processes are important as they can be used to represent noise in statistical models on spaces for which standard parametric probability distributions do not exist. We consider particles undergoing Brownian motion in some low dimensional stratified spaces, and obtain analytic solutions to the heat equation specifying the distribution of particles. These solutions play the role of prototypical distributions for studying behaviour near singularities. While probabilistic reasoning can be used to solve the heat equation in some straightforward cases, more generally we construct solutions from eigenfunctions of the Laplacian. Specifically, we solve the heat equation on: open books; two-dimensional cones; the Petersen graph with unit edge length; and the cone of this graph which corresponds to a space of evolutionary trees.     

Water-induced effects of hydrogen adsorption on Ru(001)

Adsorption and desorption of hydrogen and coadsorption of H₂ with H₂O and CO over Ru(00l) surface have been studied under UHV conditions using the technique of TDS. Surface hydrogen interacts with adsorbed water resulting in an additional desorption state at 510 K which is not easily displaced by CO, but the total number of adsorption sites for hydrogen adsorption is independent of the amount of H₂O predosed at room temperature. Hydrogen adsorption is blocked easily by CO dose (more than 0.5 L) or a small amount of O(a) formed from dissociative adsorption of water, and adsorbed hydrogen formed in the absence of significant water is easily displaced by CO dose at even room temperature.     

Mössbauer studies of illites and heat-treated illite as related to coal-conversion processes

Mossbauer spectra of iron species in the following illites were studied: Grundite, Fithian, Minerva, and New Albany. Spectra of samples of Fithian illite heated at temperatures of 225, 700, and 1000 °C were also obtained. Analyses of these spectra provide Mössbauer parameter values of iron species in the illites and heat-transformed illite for comparison with similar species found in coals containing illites and in coal process residues derived from them. The illites contain both ferric and ferrous species. Mössbauer parameters for one of the ferric species, designated M(2), are virtually the same as those of pyrite. The two species are therefore difficult to distinguish from one another. Values of the concentration of pyrite in coals and shales may be inflated if the pyrite concentration is measured by Mössbauer spectroscopy. Mössbauer spectra of the heat-treated illite samples reveal changes in iron distribution, principally at the 700 and 1000 °C levels, where there are found three and six different iron species respectively. These changes are accompanied by reduction of ferric to ferrous iron. This process should be integrated into any assessment of the iron chemistry which accompanies coal-conversion processes.