Recent Advances in Hierarchical Reinforcement Learning

Reinforcement learning is bedeviled by the curse of dimensionality: the number of parameters to be learned grows exponentially with the size of any compact encoding of a state. Recent attempts to combat the curse of dimensionality have turned to principled ways of exploiting temporal abstraction, where decisions are not required at each step, but rather invoke the execution of temporally-extended activities which follow their own policies until termination. This leads naturally to hierarchical control architectures and associated learning algorithms. We review several approaches to temporal abstraction and hierarchical organization that machine learning researchers have recently developed. Common to these approaches is a reliance on the theory of semi-Markov decision processes, which we emphasize in our review. We then discuss extensions of these ideas to concurrent activities, multiagent coordination, and hierarchical memory for addressing partial observability. Concluding remarks address open challenges facing the further development of reinforcement learning in a hierarchical setting.     

The liquidus surface and tie-lines in the iron-cobalt-sulfur system between 1473 and 1623 K

The liquidus surface and tie-lines in the iron-cobalt-sulfur ternary system have been determined between 1473 and 1623 K. The experiments were conducted by equilibrating the liquid sulfide phase with the metallic alloy phase. The liquid sulfide phase was sampled and chemically analyzed. The alloy phase was analyzed by electron microprobe. Combining the present results with the available literature data, the thermodynamic properties of this system were calculated.     

A simplified design strategy for mapping image processing algorithms on a SIMD torus

It is proposed to enhance and simplify the programming of a two dimensional (2-D) torus (and mesh) connected SIMD array of simple processing elements (PEs) by introducing two dedicated communication registers in each PE. A new SIMD algorithm to transpose a matrix using only two buffers at each PE is described. A method is proposed to effectively realize large number of arbitrary, one-to-one, personalized, and concurrent communication between the PEs, by suitably repeating the matrix transpose algorithm. Implementation of several image processing tasks of shift-variant nature, such as hough transform, histogram, median filters, which involve such communication, is enhanced by this approach. The dynamic behavior of such a SIMD implementation is data independent, unlike the ones that employ greedy methods for handling the overall communication. This feature facilitates coordinated use of several independently operating SIMD meshes in a newly emerging computer vision paradigm known as multiview image-sequence analysis (MVISA) for 3-D perception of unstructured dynamic scenes.     

Chloride metallurgy: PGM recovery and titanium dioxide production

This paper examines in detail the thermodynamics and application of chloride metallurgy for the extraction of precious metals, such as gold and silver, and platinum-group metals. The advantages with regard to the solubilities of metal ion species and their reduction potentials in chloride media are discussed with examples. The use of chloride media for the extraction of platinum-group metals from spent autocatalysts and for the production of high-purity pigment-grade TiO₂ and titanium metal from ilmenite feed stocks is discussed in the case studies provided. For more information, contact V.I. Lakshmanan, Process Research Ortech, 2395 Speakman Dr., Mississauga, ON, L5K 1B3 Canada; (905) 822-4941; fax (905) 822-9537.     

Morphological Forms of α-Alumina Particles Synthesized in 1,4-Butanediol Solution

Phase-pure, monodispersed, hexagonal plates of single-crystal α-alumina (∼ 2 μm wide and ∼0.5 μm thick) have been prepared via precipitation by treating an aluminum hydrous oxide precursor in 1,4-butanediol at 300°C under autogenous vapor pressure. Present work shows that KOH is the only reagent that precipitates an aluminum hydrous oxide precursor suitable to synthesize α-alumina in 1,4-butanediol solution. In contrast, the use of NaOH or NH₄OH as the precipitating reagent for the precursor material does not yield the alpha phase. The solution pH at which the precursor materials are precipitated is also a critical factor for the formation of α-Al₂O₃. Phase-pure α-alumina powders were also only synthesized from the aluminum hydrous oxide precursors precipitated in the pH range from 10 to 10.5. The results of X-ray diffraction and scanning electron microscopy indicate that longer reaction times promote the phase transformation from the intermediate boehmite phase to α-alumina. The complete transformation from boehmite to α-alumina requires reaction times of about 12 h.     

Anionic Clays as Potential Slow-Release Fertilizers: Nitrate Ion Exchange

Several nitrate containing anionic clays were synthesized at different temperatures and the kinetics of NO₃ ^– release were determined to test their suitability as slow-release N fertilizers. A sample (Mg:Al = 2:1) synthesized at 60°C with smaller particle size released 75, 86 and 100% of its NO₃ ^– in 1, 3 and 7 days, respectively when equilibrated with a simulated soil solution. On the other hand, the 175°C/2 hrs sample with larger particle size released 65, 77 and 84% of its nitrate in 1, 3 and 7 days, respectively. Another anionic clay (synthesized at 175°C/24 hrs) of higher charge density (Mg:Al = 2:1) containing NO₃ ^– was equilibrated with a 0.012 N NaCl or Na₂CO₃ to test the role of different anions in releasing the NO₃ ^– anion from the interlayers. The results showed that Cl^– released more NO₃ ^– than did CO₃ ^2– from this anionic clay after all the treatment times probably as a result of the CO₃ ^2– anion blocking the release of NO₃ ^– from the interior of the crystals. When a lower charge density (Mg:Al = 3:1) sample (synthesized at 175°C/48 hrs) was equilibrated with 0.02N solution of anions the release of nitrate was as follows: Cl^– < F^– < SO₄ ⁼ CO₃ ^2–. These results suggest that the divalent SO₄ ⁼ and CO₃ ^2– anions are more effective in the release of NO₃ ^– from this lower charge density anionic clay. Time-resolved structural analysis of NO₃ ^– exchange with CO₃ ^2– in the above anionic clay using synchrotron x-ray diffraction showed that ion exchange is rapid because of small crystal size and lower charge density. Thus the release of NO₃ ^– from anionic clays is an interplay among the type of anions present in soil solution, their concentration, pH of soil solution, the charge density and crystal size of anionic clay etc.     

Porous mullite ceramics through diphasic gels of large boehmite and small silica particles

Mullite formation process has been studied in stoichiometric mullite (3Al₂O₃·2SiO₂) diphasic gel containing large boehmite (1 m) and small silica (10 nm) particles. It has been found that initial mullitization did not take place inside the silica phase (cristobalite), but took place in the defect -alumina phase. -alumina was stabilized by silica when the temperature was below 1350°C. At temperatures above 1350°C, mullite crystallized directly. It was suggested that silica diffused into the pores (<1.8 nm) of -alumina and prevented the collapse of -alumina pore structure. On the other hand, when silica was not present, the pore structure of -alumina collapsed and -alumina crystallized at 1100°C. Porous mullite ceramics were obtained by using special diphasic gels containing large boehmite and small silica particles.     

Dehydration of tetrahydrofuran by pervaporation using crosslinked PVA/PEI blend membranes

Dense blend membranes were prepared by blending hydrophilic polymers poly(vinyl alcohol) (PVA) and poly(ethyleneimine) (PEI), which were then crosslinked by glutaraldehyde (GA) in a mixture of solvents under the catalysis of hydrochloric acid (HCl) for the dehydration of tetrahydrofuran (THF) by pervaporation. The effect of experimental parameters such as feed water concentration, permeate pressure, and membrane thicknesses on permeate parameters, i.e., flux and selectivity were determined with feed water concentration less than 40 wt %. The membranes were found to have good potential for breaking the azeotrope of 94 wt % THF with a flux of 1.072 and 0.376 kg/m² h for plane PVA/PEI and crosslinked PVA/PEI blend membrane, which exhibited high selectivity of 156 and 579 respectively. Selectivity was found to improve with decreasing feed water concentration and increasing membrane thickness, whereas flux decreased correspondingly. High permeate pressure causes a reduction in both flux and selectivity. These effects were clearly elucidated with the aid of the known relationship among plasticization effect, degree of swelling, permeate pressure, and feed water concentration. These blend membranes were also subjected to sorption studies to evaluate the extent of interaction and degree of swelling in pure as well as binary feed mixtures. Further ion exchange capacity studies were carried out for all the crosslinked and uncrosslinked membranes to determine the total number of interacting groups present in the membranes. © 2006 Wiley Periodicals, Inc. J Appl Polym Sci 102: 1152–1161, 2006     

Sequestration of Cesium and Strontium by Tobermorite Synthesized from Fly Ashes

Tobermorites were synthesized from oxides and from fly ashes under mild hydrothermal conditions. The bulk compositions of the ash-based formulations were adjusted to give the same [Ca]/[Si + Al] ratio as for the oxides. Selective Cs and Sr exchange with these tobermorites was measured. Compared to those prepared from the oxides and from class C fly ash, the tobermorites obtained from class F fly ash exhibited superior Cs and Sr selective properties. This is apparently a result of more extensive Al incorporation with class F fly ash. For example, tobermorite synthesized from class F fly ash exhibited a cesium K _d (mL/g) of 3826 while those prepared from oxides and from class C fly ash showed K _d values of 1112 and 796, respectively, under the same conditions. These results suggest that fly ash waste materials can be converted into tobermorite, which can serve as a resource in the separation, immobilization, and disposal of radioactive species such as Cs and Sr.