A study of Q-learning considering negative rewards

In the reinforcement learning system, the agent obtains a positive reward, such as 1, when it achieves its goal. Positive rewards are propagated around the goal area, and the agent gradually succeeds in reaching its goal. If you want to avoid certain situations, such as dangerous places or poison, you might want to give a negative reward to the agent. However, in conventional Q-learning, negative rewards are not propagated in more than one state. In this article, we propose a new way to propagate negative rewards. This is a very simple and efficient technique for Q-learning. Finally, we show the results of computer simulations and the effectiveness of the proposed method.     

Randomized constraint solvers: a comparative study

The complexity of constraints is a major obstacle for constraint-based software verification. Automatic constraint solvers are fundamentally incomplete: input constraints often build on some undecidable theory or some theory the solver does not support. This paper proposes and evaluates several randomized solvers to address this issue. We compared the effectiveness of a symbolic solver (CVC3), a random solver, two heuristic search solvers, and seven hybrid solvers (i.e. mix of random, symbolic, and heuristic solvers). We evaluated the solvers on a benchmark generated with a concolic execution of 9 subjects. The performance of each solver was measured by its precision, which is the fraction of constraints that the solver can find solution out of the total number of constraints that some solver can find solution. As expected, symbolic solving subsumes the other approaches for the 4 subjects that only generate decidable constraints. For the remaining 5 subjects, which contain undecidable constraints, the hybrid solvers achieved the highest precision (fraction of constraints that a solver can find a solution out of the total number of satisfiable constraints). We also observed that the solvers were complementary, which suggests that one should alternate their use in iterations of a concolic execution driver.     

A Network Architecture with High Availability for Real-time Premium Traffic over the Internet

Real-time traffic will be a predominant traffic type in the next generation networks, and networks with 100% reliability and availability will be required by real-time premium traffic. It is believed that QoS guarantees could be better provided by connection oriented networks such as Multi Protocol Label Switching (MPLS). These connection oriented networks are more vulnerable to network failure. Conventional path protection methods perform re-routing to cope with this. However, re-routing always causes packet losses and results in service outage. These losses are bursty in nature and highly degrade the QoS of the real-time premium traffic. Thus, 100% availability cannot be achieved by conventional methods. The novel path protection proposed in this paper recovers the bursty packet losses due to re-routing by using forward error correction (FEC) path. Therefore, it can provide network architecture with no service outage for such traffic. The numerical results show that the proposed method can achieve a very high availability for real-time premium traffic in future IP/MPLS networks.

Unravel four hairpins!

DNA machines consisting of consecutive hairpins, which we have previously described, have various potential applications in DNA computation. In the present study, a 288-base DNA machine containing four consecutive hairpins was successfully constructed by ligation and PCR. PAGE and fluorescence spectroscopy experiments verified that all four hairpins were successfully opened by four opener oligomers, and that hairpin opening was dependent on the proper openers added in the correct order. Quantitative analysis of the final results by fluorescence spectroscopy indicated that all four hairpins were open in about 1/4 to 1/3 of the DNA machines.     

Cooperation of categorical and behavioral learning in a practical solution to the abstraction problem

Real robots should be able to adapt autonomously to various environments in order to go on executing their tasks without breaking down. They achieve this by learning how to abstract only useful information from a huge amount of information in the environment while executing their tasks. This paper proposes a new architecture which performs categorical learning and behavioral learning in parallel with task execution. We call the architectureSituation Transition Network System (STNS). In categorical learning, it makes a flexible state representation and modifies it according to the results of behaviors. Behavioral learning is reinforcement learning on the state representation. Simulation results have shown that this architecture is able to learn efficiently and adapt to unexpected changes of the environment autonomously. Atsushi Ueno, Ph.D.: He is a research associate in the Artificial Intelligence Laboratory at the Graduate School of Information Science at the Nara Institute of Science and Technology (NAIST). He received the B.E., the M.E., and the Ph.D. degrees in aeronautics and astronautics from the University of Tokyo in 1991, 1993, and 1997 respectively. His research interest is robot learning and autonomous systems. He is a member of Japan Association for Artificial Intelligence (JSAI). Hideaki Takeda, Ph.D.: He is an associate professor in the Artificial Intelligence Laboratory at the Graduate School of Information Science at the Nara Institute of Science and Technology (NAIST). He received his Ph.D. in precision machinery engineering from the University of Tokyo in 1991. He has conducted research on a theory of intelligent computer-aided design systems, in particular experimental study and logical formalization of engineering design. He is also interested in multiagent architectures and ontologies for knowledge base systems.     

Japanese/English Cross-Language Information Retrieval: Exploration of Query Translation and Transliteration

Cross-language information retrieval (CLIR), where queriesand documents are in different languages, has of late become one ofthe major topics within the information retrieval community. Thispaper proposes a Japanese/English CLIR system, where we combine aquery translation and retrieval modules. We currently target theretrieval of technical documents, and therefore the performance of oursystem is highly dependent on the quality of the translation oftechnical terms. However, the technical term translation is stillproblematic in that technical terms are often compound words, and thusnew terms are progressively created by combining existing basewords. In addition, Japanese often represents loanwords based on itsspecial phonogram. Consequently, existing dictionaries find itdifficult to achieve sufficient coverage. To counter the firstproblem, we produce a Japanese/English dictionary for base words, andtranslate compound words on a word-by-word basis. We also use aprobabilistic method to resolve translation ambiguity. For the secondproblem, we use a transliteration method, which corresponds wordsunlisted in the base word dictionary to their phonetic equivalents inthe target language. We evaluate our system using a test collectionfor CLIR, and show that both the compound word translation andtransliteration methods improve the system performance.     

MOSAIC for multiple-reward environments

Reinforcement learning (RL) can provide a basic framework for autonomous robots to learn to control and maximize future cumulative rewards in complex environments. To achieve high performance, RL controllers must consider the complex external dynamics for movements and task (reward function) and optimize control commands. For example, a robot playing tennis and squash needs to cope with the different dynamics of a tennis or squash racket and such dynamic environmental factors as the wind. In addition, this robot has to tailor its tactics simultaneously under the rules of either game. This double complexity of the external dynamics and reward function sometimes becomes more complex when both the multiple dynamics and multiple reward functions switch implicitly, as in the situation of a real (multi-agent) game of tennis where one player cannot observe the intention of her opponents or her partner. The robot must consider its opponent's and its partner's unobservable behavioral goals (reward function). In this article, we address how an RL agent should be designed to handle such double complexity of dynamics and reward. We have previously proposed modular selection and identification for control (MOSAIC) to cope with nonstationary dynamics where appropriate controllers are selected and learned among many candidates based on the error of its paired dynamics predictor: the forward model. Here we extend this framework for RL and propose MOSAIC-MR architecture. It resembles MOSAIC in spirit and selects and learns an appropriate RL controller based on the RL controller's TD error using the errors of the dynamics (the forward model) and the reward predictors. Furthermore, unlike other MOSAIC variants for RL, RL controllers are not a priori paired with the fixed predictors of dynamics and rewards. The simulation results demonstrate that MOSAIC-MR outperforms other counterparts because of this flexible association ability among RL controllers, forward models, and reward predictors.     

Construction of a brain–machine hybrid system to evaluate adaptability of an insect

Insects perform adaptive behavior according to changing environmental conditions using comparatively small brains. Because adaptability is generated through the relationship among brain, body and environment, it is necessary to examine how a brain works under these conditions. In this study, to understand neural processing involved in adaptive behavior, we constructed a brain–machine hybrid system using motor signals related to the steering behavior of the male silkworm moth for controlling a two-wheeled mobile robot. We developed this hybrid system according to the following steps. (1) We selected steering signals corresponding to walking direction that were activated during neck swinging induced by optic flow and pheromone stimuli. (2) To control a robot by neural activity, we implemented a spike-behavior conversion rule such that frequency of the left and right neck motor neurons’ spikes was linearly converted into rotation of the wheels. (3) For electrophysiological multi-unit recordings on a robot, we developed small amplifiers. Using this hybrid system, we could observe the programmed behavioral pattern and orientation toward a pheromone source. Moreover, we compared the orientation behavior of moths and that of the hybrid system at different pheromone stimulus frequencies. From these experiments, we concluded that we could reconstruct silkworm moth behavior on the hybrid system.     

Dynamic alternation of primate response properties during trial-and-error knowledge updating

Humans and animals seek appropriate solutions to novel problems through trial-and-error (TE) actions and observation of their outcomes. Once an individual has obtained the knowledge (rule) to solve a problem, knowledge-based (KB) actions may be applied in a stereotypical manner. Solutions can thus be based on TE or KB actions. To characterize this learning process at the behavioral level, we developed a new cognitive task for a laboratory monkey (Macaca fuscata) to perform. In this task, a search array consisting of six elements of different colors was presented, one of which was the behaviorally relevant target. The target color was changed unpredictably with no instruction or signal, requiring the monkey to use a TE search strategy to find the target color. We found that once the monkey identified the relevant color by chance after a color change, correct performance increased in a step-like manner and at the same time, other response properties (reaction time and color-choice tendency) also changed discontinuously. These step-like alternations in behavioral performance may be attributed to the subject’s switching between TE and KB search strategies in the two phases. The present study has therefore provided behavioral evidence for the timing and manner of switching between search strategies during the process of updating knowledge.