Reward revision and the average reward markov decision process

Summary We integrate two numerical procedures for solving the average reward Markov decision process (MDP), standard successive approximations and modified policy iteration with reward revision. Reward revision is the process of revising the reward structure of a second, more computationally desirable MDP so as to produce, in the limit, an optimality equation having a fixed point identical to that associated with the original MDP. A numerical study indicates that for MDP's having a non-sparse transition structure with a small number of relatively large entries per row, the addition of reward revision can have significant computational benefits.

---

Zusammenfassung Zur Lösung Markovscher Entscheidungsprozesse (MDP) mit Durchschnitts-Kriterium werden zwei numerische Verfahren, nämlich sukzessive Approximation und modifizierte Politik-Iteration, mit einer Transformation, der sogenannten 'Reward-Revision, kombiniert. Bei dieser Transformation werden die Übergangswahrscheinlichkeiten so abgeändert (ausgedünnt), daß das neue Modell sich numerisch günstiger verhält. Dazu müssen die einstufigen Erträge so revidiert werden, daß die Optimalitäts-Gleichung des neuen Modells im Limes mit der des ursprünglichen übereinstimmt. Numerische Untersuchungen zeigen, daß für MDP mit stark besetzten Übergangsmatrizen, bei denen nur an wenigen Stellen je Zeile große Werte stehen, die Anwendung von 'Reward Revision zu wesentlichen Einsparungen an Rechenaufwand führen kann.

---

---

Research supported by NSF Grant ECS-8319355     

Biflavonoid-Induced Disruption of Hydrogen Bonds Leads to Amyloid-β Disaggregation

Deposition of amyloid β (Aβ) fibrils in the brain is a key pathologic hallmark of Alzheimer’s disease. A class of polyphenolic biflavonoids is known to have anti-amyloidogenic effects by inhibiting aggregation of Aβ and promoting disaggregation of Aβ fibrils. In the present study, we further sought to investigate the structural basis of the Aβ disaggregating activity of biflavonoids and their interactions at the atomic level. A thioflavin T (ThT) fluorescence assay revealed that amentoflavone-type biflavonoids promote disaggregation of Aβ fibrils with varying potency due to specific structural differences. The computational analysis herein provides the first atomistic details for the mechanism of Aβ disaggregation by biflavonoids. Molecular docking analysis showed that biflavonoids preferentially bind to the aromatic-rich, partially ordered N-termini of Aβ fibril via the π–π interactions. Moreover, docking scores correlate well with the ThT EC₅₀ values. Molecular dynamic simulations revealed that biflavonoids decrease the content of β-sheet in Aβ fibril in a structure-dependent manner. Hydrogen bond analysis further supported that the substitution of hydroxyl groups capable of hydrogen bond formation at two positions on the biflavonoid scaffold leads to significantly disaggregation of Aβ fibrils. Taken together, our data indicate that biflavonoids promote disaggregation of Aβ fibrils due to their ability to disrupt the fibril structure, suggesting biflavonoids as a lead class of compounds to develop a therapeutic agent for Alzheimer’s disease.     

The effects of bleached and unbleached rosemary oleoresins on light-sensitized oxidation of soybean oil

Bleached and unbleached forms of a rosemary oleoresin (RO) in stripped and nonstripped soybean oil behave both as antioxidant and prooxidant in a light-induced oxidative system. At 0.02 and 0.05% levels, RO had the greatest antioxidant activity, while at 0.01 and 0.5% levels it had the highest prooxidant activity in both stripped and nonstripped soybean oil. Treatment of both soybean oil systems with tertiary butylhydroquinone controlled light-induced oxidation of the oil better than did the oleoresin treatments. The prooxidant activity of the 0.5% RO level was probably due to an excess of prooxidant components being carried into the oil at that level, whereas the reduced antioxidant activity at 0.01% was probably due to the low initial level of active antioxidant components being added to the oil. Published as Journal Series No. 10072, Nebraska Agricultural Research Division, Department of Food Science and Technology, University of Nebraska, Lincoln, NE 68583-0919.     

Stability of Aqueous α-Al2O3 Suspensions with Poly(methacrylic acid) Polyelectrolyte

Stability of aqueous α-Al₂O₃ suspensions with Na⁺ salt of poly(methacrylic acid) (PMAA-Na) polyelectrolyte was studied as a function of pH. At a given pH, the transition from the flocculated to the dispersed state corresponded to the adsorption saturation limit of the powders by the PMAA. As the pH was decreased, the adsorption saturation limit increased until insolubility and charge neutralization of the PMAA was approached. The critical amount of PMAA required to achieve stability is outlined in a stability map.     

THE ROLE OF LATTICE OXYGEN IN THE DYNAMIC BEHAVIOR OF OXIDE CATALYSTS

The lattice of an oxide catalyst used for oxidation reactions can act as a reservoir for oxygen, storing and releasing it for reactions at the catalyst surface under appropriate conditions. The implication of this oxygen storage property of an oxide catalyst on its dynamic response characteristics has been investigated through an experimental study of 2-butene oxidation over vanadium oxide as a model reaction. Isothermal reaction rate measurements in a differential reactor and nonisothermal studies in a single pellet reactor have been carried out. Following a step increase in the feed butene concentration, isothermal reaction rate overshoot and pellet temperature overshoot were observed. These observations could be modelled in a qualitatively correct way by a very simple model accounting for the participation of lattice oxygen in the catalytic reactions under dynamic conditions. It is demonstrated through model simulations that the ignition characteristics of a catalyst pellet are significantly affected by the participation of the lattice oxygen, when steady state multiplicity is present.     

Shape memory polymers based on uniform aliphatic urethane networks

Aliphatic urethane polymers have been synthesized and characterized, using monomers with high molecular symmetry, to form amorphous networks with very uniform supermolecular structures, which can be used as photo‐thermally actuable shape memory polymers (SMPs). The monomers used include hexamethylene diisocyanate (HDI), trimethylhexamethylenediamine (TMHDI), N,N,N′,N′‐tetrakis(hydroxypropyl)ethylenediamine (HPED), triethanolamine (TEA), and 1,3‐butanediol (BD). The new polymers were characterized by solvent extraction, NMR, XPS, UV/VIS, DSC, DMTA, and tensile testing. The resulting polymers were found to be single phase amorphous networks with very high gel fraction, excellent optical clarity, and extremely sharp single glass transitions in the range of 34–153°C. Thermomechanical testing of these materials confirms their excellent shape memory behavior, high recovery force, and low mechanical hysteresis (especially on multiple cycles), effectively behaving as ideal elastomers above T_g. We believe these materials represent a new and potentially important class of SMPs, and should be especially useful in applications such as biomedical microdevices. © 2007 Wiley Periodicals, Inc. J Appl Polym Sci, 2007     

Manufacturing Processes for Long-Life Gas Turbines

Dual-alloy turbine wheels produced by solid-state diffusion bonding of vacuum investment cast blade rings of one superalloy to preconsolidated powder metal hubs of a second superalloy have the long cyclic lives characteristic of wrought or powder superalloys combined with the high creep strength and net-shape blades characteristic of cast superalloys. A wide variety of superalloys and turbine configurations are compatible with this technology. Improved temperature capability turbine blades and vanes of the MAR-M 247 alloy made by directional solidification casting processes are now in volume production for Garrett gas turbines. Single-crystal alloys derivative to MAR-M 247 further extend the temperature capability of turbine blades and have been successfully engine tested. These blades are produced by a relatively simple modification of the processes used to manufacture directionally solidified blades.     

Do Management Actions to Restore Rare Habitat Benefit Native Fish Conservation? Distribution of Juvenile Native Fish Among Shoreline Habitats of the Colorado River

Many management actions in aquatic ecosystems are directed at restoring or improving specific habitats to benefit fish populations. In the Grand Canyon reach of the Colorado River, experimental flow operations as part of the Glen Canyon Dam Adaptive Management Program have been designed to restore sandbars and associated backwater habitats. Backwaters can have warmer water temperatures than other habitats, and native fish, including the federally endangered humpback chub Gila cypha, are frequently observed in backwaters, leading to a common perception that this habitat is critical for juvenile native fish conservation. However, it is unknown how fish densities in backwaters compare with that in other habitats or what proportion of juvenile fish populations reside in backwaters. Here, we develop and fit multi‐species hierarchical models to estimate habitat‐specific abundances and densities of juvenile humpback chub, bluehead sucker Catostomus discobolus, flannelmouth sucker Catostomus latipinnis and speckled dace Rhinichthys osculus in a portion of the Colorado River. Densities of all four native fish were greatest in backwater habitats in 2009 and 2010. However, backwaters are rare and ephemeral habitats, so they contain only a small portion of the overall population. For example, the total abundance of juvenile humpback chub in this study was much higher in talus than in backwater habitats. Moreover, when we extrapolated relative densities based on estimates of backwater prevalence directly after a controlled flood, the majority of juvenile humpback chub were still found outside of backwaters. This suggests that the role of controlled floods in influencing native fish population trends may be limited in this section of the Colorado River. Copyright © 2014 John Wiley & Sons, Ltd.     

Multi-rate model predictive control of particle size distribution in a semibatch emulsion copolymerization reactor

A multi-rate model predictive controller algorithm is presented for the in-batch closed-loop control of the full particle size distribution (PSD) in a semibatch emulsion copolymerization system. The lack of frequent measurements of the PSD and the measurement delay of these measurements are addressed through the use of frequent density measurements from which the current conditions of the system are estimated. The high dimensionality of the discretized full PSD is reduced by the use of model order reduction based on principal component analysis. This method effectively reduces the size of the problem while preserving the main characteristics of the population balance system. Disturbances that perturb the surfactant and monomer amounts inside the semibatch vinyl acetate–butyl acrylate reactor are considered to demonstrate the performance of the proposed control algorithm.     

Effect of pressure on three catalytic partial oxidation reactions at millisecond contact times

The effects of pressure on reactant conversion and product selectivities in three catalytic oxidation systems have been examined at pressures between 1 and > 5 atm. Reaction was sustained autothermally near adiabatic operating conditions at temperatures of 1000°C with residence times over the noble metal catalysts between 10^–4 and 10^–2 s. The three systems investigated were (1) HCN synthesis over Pt-10% Rh gauze catalysts, (2) methane oxidation to synthesis gas (CO and H₂) over rhodium-coated monoliths, and (3) ethane conversion to ethylene over platinum-coated monoliths. We find that selectivities in all three reactions do not change dramatically with approximately a five-fold increase in pressure. This strongly suggests that free radical homogeneous chain reactions are not significant in these processes and that they can be operated reliably above atmospheric pressure. For the synthesis of HCN over Pt-10% Rh gauzes, the selectivity to HCN can be maintained above 0.75 at pressures up to 5.5 atm. Selectivities to synthesis gas (CO and H₂) from a methane-air mixture over a Rh-coated foam monolith at pressures up to 5.5 atm were maintained above 0.90. Over a Pt-coated foam monolith, the selectivity to ethylene from ethane-air and ethane-O₂ mixtures was independent of pressure up to 6.5 atm and conversion rose slightly although it was necessary to maintain constant velocity and residence time over the catalyst to avoid carbon formation.This research was supported by DOE under Grant No. DE-FG02-88ER13878.