A study of Q-learning considering negative rewards

In the reinforcement learning system, the agent obtains a positive reward, such as 1, when it achieves its goal. Positive rewards are propagated around the goal area, and the agent gradually succeeds in reaching its goal. If you want to avoid certain situations, such as dangerous places or poison, you might want to give a negative reward to the agent. However, in conventional Q-learning, negative rewards are not propagated in more than one state. In this article, we propose a new way to propagate negative rewards. This is a very simple and efficient technique for Q-learning. Finally, we show the results of computer simulations and the effectiveness of the proposed method.     

Unravel four hairpins!

DNA machines consisting of consecutive hairpins, which we have previously described, have various potential applications in DNA computation. In the present study, a 288-base DNA machine containing four consecutive hairpins was successfully constructed by ligation and PCR. PAGE and fluorescence spectroscopy experiments verified that all four hairpins were successfully opened by four opener oligomers, and that hairpin opening was dependent on the proper openers added in the correct order. Quantitative analysis of the final results by fluorescence spectroscopy indicated that all four hairpins were open in about 1/4 to 1/3 of the DNA machines.     

Cooperation of categorical and behavioral learning in a practical solution to the abstraction problem

Real robots should be able to adapt autonomously to various environments in order to go on executing their tasks without breaking down. They achieve this by learning how to abstract only useful information from a huge amount of information in the environment while executing their tasks. This paper proposes a new architecture which performs categorical learning and behavioral learning in parallel with task execution. We call the architectureSituation Transition Network System (STNS). In categorical learning, it makes a flexible state representation and modifies it according to the results of behaviors. Behavioral learning is reinforcement learning on the state representation. Simulation results have shown that this architecture is able to learn efficiently and adapt to unexpected changes of the environment autonomously. Atsushi Ueno, Ph.D.: He is a research associate in the Artificial Intelligence Laboratory at the Graduate School of Information Science at the Nara Institute of Science and Technology (NAIST). He received the B.E., the M.E., and the Ph.D. degrees in aeronautics and astronautics from the University of Tokyo in 1991, 1993, and 1997 respectively. His research interest is robot learning and autonomous systems. He is a member of Japan Association for Artificial Intelligence (JSAI). Hideaki Takeda, Ph.D.: He is an associate professor in the Artificial Intelligence Laboratory at the Graduate School of Information Science at the Nara Institute of Science and Technology (NAIST). He received his Ph.D. in precision machinery engineering from the University of Tokyo in 1991. He has conducted research on a theory of intelligent computer-aided design systems, in particular experimental study and logical formalization of engineering design. He is also interested in multiagent architectures and ontologies for knowledge base systems.     

Japanese/English Cross-Language Information Retrieval: Exploration of Query Translation and Transliteration

Cross-language information retrieval (CLIR), where queriesand documents are in different languages, has of late become one ofthe major topics within the information retrieval community. Thispaper proposes a Japanese/English CLIR system, where we combine aquery translation and retrieval modules. We currently target theretrieval of technical documents, and therefore the performance of oursystem is highly dependent on the quality of the translation oftechnical terms. However, the technical term translation is stillproblematic in that technical terms are often compound words, and thusnew terms are progressively created by combining existing basewords. In addition, Japanese often represents loanwords based on itsspecial phonogram. Consequently, existing dictionaries find itdifficult to achieve sufficient coverage. To counter the firstproblem, we produce a Japanese/English dictionary for base words, andtranslate compound words on a word-by-word basis. We also use aprobabilistic method to resolve translation ambiguity. For the secondproblem, we use a transliteration method, which corresponds wordsunlisted in the base word dictionary to their phonetic equivalents inthe target language. We evaluate our system using a test collectionfor CLIR, and show that both the compound word translation andtransliteration methods improve the system performance.     

破伤风毒素的提纯及形成离子通道的特性

应用硫酸铵分段沉淀和 Ultrogei AcA_(34)凝胶过滤对破伤风毒进行了提纯,并对形成的离子通道在中毒作用中的功能进行了讨论。提纯的毒素与抗粗制毒素血清形成一条沉淀线,于聚丙烯酰胺凝胶电泳中形成单一的蛋白带,分子量为15万,等电点为5.95,毒力为1～3×10~4MLD/μg 蛋白。给小鼠注射低剂量毒素引起迟发性强直性痉挛;注射高剂量毒素引起快速的类肉毒毒素的松弛性麻痹,直至死亡。应用 Patch clamp 技术,破伤风毒素于人工磷脂膜上形成离子通道。只有当膜两侧存有 pH 梯度时毒素才形成离子通道,因此是 pH 依赖的。通道的电导系数为9 PS。     

Strength Analysis of Yttria-Stabilized Tetragonal Zirconia Polycrystals

Tensile strength of Y₂O₃-stabilized ZrO₂ polycrystals (Y-TZP) was measured by a newly developed tensile testing method with a rectangular bar. The tensile strength of Y-TZP was lower than that of the three-point bend strength, and the shape of the tensile strength distribution was quite different from that of the three-point bend strength distribution. It was difficult to predict the distribution curve of the tensile strength using the data of the three-point bend strength by one-modal Weibull distribution. The distribution of the tensile strength was analyzed by two- or three-model Weibull distribution coupled with an analysis of fracture origins. The distribution curve of the three-point bend strength which was estimated by multimodal Weibull distribution agreed favorably with that of the measured three-point bend strength values. A two-modal Weibull distribution function was formulated approximately from the distributions of the tensile and three-point bend strengths, and the estimated two-modal Weibull distribution function for the four-point bend strength agreed well with the measured four-point bend strength.     

Construction of a brain–machine hybrid system to evaluate adaptability of an insect

Insects perform adaptive behavior according to changing environmental conditions using comparatively small brains. Because adaptability is generated through the relationship among brain, body and environment, it is necessary to examine how a brain works under these conditions. In this study, to understand neural processing involved in adaptive behavior, we constructed a brain–machine hybrid system using motor signals related to the steering behavior of the male silkworm moth for controlling a two-wheeled mobile robot. We developed this hybrid system according to the following steps. (1) We selected steering signals corresponding to walking direction that were activated during neck swinging induced by optic flow and pheromone stimuli. (2) To control a robot by neural activity, we implemented a spike-behavior conversion rule such that frequency of the left and right neck motor neurons’ spikes was linearly converted into rotation of the wheels. (3) For electrophysiological multi-unit recordings on a robot, we developed small amplifiers. Using this hybrid system, we could observe the programmed behavioral pattern and orientation toward a pheromone source. Moreover, we compared the orientation behavior of moths and that of the hybrid system at different pheromone stimulus frequencies. From these experiments, we concluded that we could reconstruct silkworm moth behavior on the hybrid system.     

Dynamic alternation of primate response properties during trial-and-error knowledge updating

Humans and animals seek appropriate solutions to novel problems through trial-and-error (TE) actions and observation of their outcomes. Once an individual has obtained the knowledge (rule) to solve a problem, knowledge-based (KB) actions may be applied in a stereotypical manner. Solutions can thus be based on TE or KB actions. To characterize this learning process at the behavioral level, we developed a new cognitive task for a laboratory monkey (Macaca fuscata) to perform. In this task, a search array consisting of six elements of different colors was presented, one of which was the behaviorally relevant target. The target color was changed unpredictably with no instruction or signal, requiring the monkey to use a TE search strategy to find the target color. We found that once the monkey identified the relevant color by chance after a color change, correct performance increased in a step-like manner and at the same time, other response properties (reaction time and color-choice tendency) also changed discontinuously. These step-like alternations in behavioral performance may be attributed to the subject’s switching between TE and KB search strategies in the two phases. The present study has therefore provided behavioral evidence for the timing and manner of switching between search strategies during the process of updating knowledge.