AN ANALYSIS OF EXPERIENCE REPLAY IN TEMPORAL DIFFERENCE LEARNING

Temporal difference TD methods are used by reinforcement learning algorithms for predicting future rewards. This article analyzes theoretically and illustrates experimentally the effects of performing TD lambda prediction udpates backwards for a number of past experiences. More exactly, two related techniques described in the literature are examined, referred to as replayed TD and backwards TD. The former is essentially an online learning method which performs at each time step a regular TD 0 update, and then replays updates backwards for a number of previous states. The latter operates in offline mode, after the end of a trial updating backwards the predictions for all visited states. They are both shown to be approximately equivalent to TD lambda with variable lambda values selected in a particular way. This is true even if they perform only TD 0 updates. The experimental results show that replayed TD 0 is competitive to TD lambda with regard to learning speed and quality.     

An empirical analysis of heuristics for solving the two-machine flow shop problem with job release times

The theoretical analysis of heuristics for solving intractable optimization problems has many well-known drawbacks. Constructed instances demonstrating an exceptionally poor worst-case performance of heuristics are typically too peculiar to occur in practice. Theoretical results on the average-case performance of most heuristics could not be established due to the difficulty with the use of probabilistic analysis. Moreover, the heuristics for which some type of asymptotic optimality has been proven are likely to perform questionably in practice. The purpose of this paper is to confront known theoretical results with our empirical results concerning heuristics for solving the strongly NP-hard problem of minimizing the makespan in a two-machine flow shop with job release times. The heuristics' performance is examined with respect to their average and maximum relative errors, as well as their optimality rate, that is, the probability of being optimal. In particular, this allows us to observe that the asymptotic optimality rate of so called “almost surely asymptotically optimal” heuristic can be zero. We also present a new heuristic with short worst-case running time and statistically prove that it outperforms all heuristics known so far. However, our empirical experiments reveal that the heuristic is on average slower that its competitors with much longer worst-case running times.     

Segmentation of microscope images of living cells

This paper describes a segmentation method combining a texture based technique with a contour based method. The technique is designed to enable the study of cell behaviour over time by segmenting brightfield microscope image sequences. The technique was tested on artificial images, based on images of living cells and on real sequences acquired from microscope observations of neutrophils and lymphocytes as well as on a sequence of MRI images. The results of the segmentation are compared with the results of the watershed and snake segmentation methods. The results show that the method is both effective and practical.

Integrated Data Management for a Fleet of Search‐and‐rescue Robots

Search‐and‐rescue operations have recently been confronted with the introduction of robotic tools that assist the human search‐and‐rescue workers in their dangerous but life‐saving job of searching for human survivors after major catastrophes. However, the world of search and rescue is highly reliant on strict procedures for the transfer of messages, alarms, data, and command and control over the deployed assets. The introduction of robotic tools into this world causes an important structural change in this procedural toolchain. Moreover, the introduction of search‐and‐rescue robots acting as data gatherers could potentially lead to an information overload toward the human search‐and‐rescue workers, if the data acquired by these robotic tools are not managed in an intelligent way. With that in mind, we present in this paper an integrated data combination and data management architecture that is able to accommodate real‐time data gathered by a fleet of robotic vehicles on a crisis site, and we present and publish these data in a way that is easy to understand by end‐users. In the scope of this paper, a fleet of unmanned ground and aerial search‐and‐rescue vehicles is considered, developed within the scope of the European ICARUS project. As a first step toward the integrated data‐management methodology, the different robotic systems require an interoperable framework in order to pass data from one to another and toward the unified command and control station. As a second step, a data fusion methodology will be presented, combining the data acquired by the different heterogenic robotic systems. The computation needed for this process is done in a novel mobile data center and then (as a third step) published in a software as a service (SaaS) model. The SaaS model helps in providing access to robotic data over ubiquitous Ethernet connections. As a final step, we show how the presented data‐management architecture allows for reusing recorded exercises with real robots and rescue teams for training purposes and teaching search‐and‐rescue personnel how to handle the different robotic tools. The system was validated in two experiments. First, in the controlled environment of a military testing base, a fleet of unmanned ground and aerial vehicles was deployed in an earthquake‐response scenario. The data gathered by the different interoperable robotic systems were combined by a novel mobile data center and presented to the end‐user public. Second, an unmanned aerial system was deployed on an actual mission with an international relief team to help with the relief operations after major flooding in Bosnia in the spring of 2014. Due to the nature of the event (floods), no ground vehicles were deployed here, but all data acquired by the aerial system (mainly three‐dimensional maps) were stored in the ICARUS data center, where they were securely published for authorized personnel all over the world. This mission (which is, to our knowledge, the first recorded deployment of an unmanned aerial system by an official governmental international search‐and‐rescue team in another country) proved also the concept of the procedural integration of the ICARUS data management system into the existing procedural toolchain of the search and rescue workers, and this in an international context (deployment from Belgium to Bosnia). The feedback received from the search‐and‐rescue personnel on both validation exercises was highly positive, proving that the ICARUS data management system can efficiently increase the situational awareness of the search‐and‐rescue personnel.     

Framework for application management with dynamic aspects J-EARS case study

The cost of computer system maintenance rises together with the increasing complexity of such systems. The use of an autonomic system architecture saves money by delegating some forms of maintenance to the systems themselves. The aim of this paper is to describe the results of creating a tool which introduces elements of adaptivity to Java applications using dynamic aspects. The impact of introducing aspects on the performance of various Application Servers is also discussed. Finally, benefits and problems arising from the use of the tool are presented, basing on sample use cases.     

Algorithms for spatial collocation pattern mining in a limited memory environment: a summary of results

Rapid growth of spatial datasets requires methods to find (semi-)automatically spatial knowledge from these sets. Spatial collocation patterns represent subsets of spatial features whose instances are frequently located together in a spatial neighborhood. In recent years, efficient methods for collocation discovery have been developed, however, none of them assume limited size of the operational memory or limited access to memory with short access times. Such restrictions are especially important in the context of the large size of the data structures required for efficient identification of collocation instances. In this work we present and compare three algorithms for collocation pattern mining in a limited memory environment. The first algorithm is based on the well-known joinless method introduced by Shekhar and Yoo. The second and third algorithms are inspired by a tree structure (iCPI-tree) presented by Wang et al. In our experimental evaluation, we have compared the efficiency of the algorithms, both on synthetic and real world datasets.     

Magnetic Properties of Nickel–Zinc Ferrite Toroids Prepared from Nanoparticles

Toroids comprised of silica-coated 10 nm diameter nickel–zinc (Ni–Fe) ferrite nanoparticles (Ni_0.5Zn_0.5Fe₂O₄) have been fabricated by careful control of both the coating process and subsequent densification by viscous sintering. A narrow processing window is identified between a maximum temperature at which the nanoparticles coarsen, losing their super-paramagnetic properties, and a lower temperature required for viscous flow densification. Key to the successful fabrication was drying and cold isostatic pressing of the silica-coated nanoparticles; other routes invariably led to cracking during either drying or sintering. The super-paramagnetic blocking temperature, the coercive field, and remanent magnetization could all be controlled over a wide range by varying the thickness of the silica coating from 1 to 15 nm. The dipole–dipole coupling distance is estimated to be 4 nm. The high-frequency (1–500 MHz) properties were sensitive to the sintering temperature as well as the thickness of the silica coating. Toroids sintered at 1000°C or less exhibited no high-frequency magnetic losses and their permeability decreased with increasing temperature, suggesting that the permeability was controlled by thermally activated magnetization relaxation.     

An integrated multichannel waveform generator for large-scale spatio-temporal stimulation of neural tissue

We present an ASIC designed for electrical stimulation of neural tissue using multielectrode arrays. The ASIC is foreseen for applications in systems that require simultaneous stimulation and recording of signals from various types of neural tissue, both in vitro and in vivo. The developed ASIC comprises 64 independent stimulation channels, which are capable to generate arbitrarily defined bipolar current or voltage waveforms, controlled in real time with time resolution of 12.5 μs and amplitude resolution of 7 bits. The amplitude range of output signal can be scaled over a very wide range, which ensures compatibility with various electrode arrays of different size and geometry. Each channel is also equipped with a stimulation artifact suppressor controlled in real time, which reduces the dead time of the system after each stimulation pulse.     

The profile of volatile compounds and polyphenols in wines produced from dessert varieties of apples

The aim of this study was to determine the influence of apple variety (Šampion, Idared and Gloster) on the polyphenol profile, volatile composition and sensory characteristics of apple wines. Apples were harvested from the orchard in Garlica Murowana (Poland) and the experiments were conducted on a laboratory scale. Statistically significant differences were detected in the chemical composition of the analyzed wines. The highest antioxidant activity was found in Šampion wines, which was associated with a relatively high concentration of chlorogenic acid and procyanidins. These samples also contained high amounts of acetaldehyde, ethyl acetate and methanol. Idared wines showed a similar polyphenol profile, but they had lower antioxidant capacity and were characterized by a high level of butanol and acetic acid. Gloster wines were distinguished from other samples by a lower concentration of polyphenols and higher concentration of fusel alcohols. During sensory evaluation, wines produced from Idared apples scored the highest value for overall quality.