Reinforcement learning using a stochastic gradient method with memory‐based learning

In this paper, a learning algorithm combining memory‐less learning and memory‐based learning is proposed for agents operating under POMDP. In the first stage of the proposed algorithm, memory‐less learning is applied. The stochastic gradient method is employed as a memory‐less learning algorithm. In the first stage, a state‐action set series that accomplishes the task is stored in memory. In the second stage, memory‐based learning is applied. In this process, only the series obtained in the first stage is used, so that this method is able to reduce significantly the amount of required memory. The proposed algorithm is applied to three simulations for comparison with the memory‐less learning algorithm. Through computer simulations, it is shown that the proposed algorithm works more effectively in POMDP than ordinary memory‐less learning. © 2010 Wiley Periodicals, Inc. Electr Eng Jpn, 173(1): 32–40, 2010; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20963     

Reconstructing clusters for preconditioned short‐term load forecasting

This paper presents a new preconditioned method for short‐term load forecasting that focuses on more accurate predicted value. In recent years, the deregulated and competitive power market increases the degree of uncertainty. As a result, more sophisticated short‐term load forecasting techniques are required to deal with more complicated load behavior. To alleviate the complexity of load behavior, this paper presents a new preconditioned model. In this paper, clustering results are reconstructed to equalize the number of learning data after clustering with the Kohonen‐based neural network. That enhances a short‐term load forecasting model at each reconstructed cluster. The proposed method is successfully applied to real data of one‐step ahead daily maximum load forecasting. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 161(1): 26–33, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20300     

CMOL implementation of spiking neurons and spike‐timing dependent plasticity

Successful implementation of spiking neural networks onto CMOS‐Molecular (CMOL) architecture has already been proposed, but the ability of dynamic learning has not yet been addressed. Here, we propose a spiking neural topology with spike‐timing‐dependent learning ability and provide its basic building blocks that are easily mapped onto CMOL architecture. The learning method modifies state of synaptic switches, using spatially and temporally local information which is available at the synapse when state modification is performed. The performance of the proposed topology is analyzed with regards to pre‐ and post‐synaptic spike timing, and simulation results are provided for a synapse with spike‐timing‐dependent plasticity properties. Furthermore, its performance as spike‐timing correlation learning and synchrony detection in a small feed‐forward network is demonstrated as a case example. Copyright © 2010 John Wiley & Sons, Ltd.     

Evolutionary reinforcement learning system with time‐varying parameters

In this paper, an evolutionary reinforcement learning system with time‐varying parameters that can learn appropriate policy in dynamical POMDPs is proposed. The proposed system has time‐varying parameters that can be adjusted by using reinforcement learning. Hence, the system can adapt to the time variation of the dynamical environment even if its variation cannot be observed. In addition, the state space of the environment is divided evolutionarily. Thus, one need not divide the state space in advance. The efficacy of the proposed system is shown by mobile robot control simulation under the environment belonging to dynamical POMDPs. The environment is the passage that has gates iterate opening and closing. © 2006 Wiley Periodicals, Inc. Electr Eng Jpn, 156(1): 54–60, 2006; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20170     

Closed‐loop iterative learning control for non‐linear systems with initial shifts

This paper is concerned with the problem of the iterative learning control with current cycle feedback for a class of non‐linear systems with well‐defined relative degree. The tracking error caused by a non‐zero initial shift is detected as extended D‐type learning algorithm is applied. The defect is overcome by adding terms including the output error, its derivatives as well as integrals. Asymptotic tracking of the final output to the desired trajectory is guaranteed. As an alternative approach, an initial rectifying action is introduced in the extended D‐type learning algorithm and shown effective to achieve the desired trajectory jointed smoothly with a transitional trajectory from the starting position. Also these algorithms with adjustable tracking interval ensure better robustness performance in the presence of initial shifts. Numerical simulation is conducted to demonstrate the theoretical results. Copyright © 2002 John Wiley & Sons, Ltd.     

Maximum‐power‐point tracking method of photovoltaic power system using a single transducer

This paper describes a novel strategy of maximum‐power‐point tracking for photovoltaic power generation systems. A unique feature of this method is the capacity to see the maximum power point using only a single transducer, that is, a Hall‐effect CT or an isolation amplifier. Output power of the photovoltaic can be estimated with an average value and ripple amplitude of the detected reactor current or the capacitor voltage. A conventional hill‐climbing method is employed to seek the maximum power point, using the output power estimated with only the current or voltage transducer. In this paper, not only a theoretical aspect of the proposed method is discussed, but also experimental results are presented to prove the feasibility of the method. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 160(1): 79– 86, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20275     

Magnetic properties of PLD‐fabricated ND‐Fe‐B film magnets and their applications to millimeter‐size motors

We succeeded in depositing 200‐ to 300‐µm‐thick Nd‐Fe‐B film magnets with (BH)_max of approximately 55 kJ/m³ on Fe substrates by the PLD method. In addition, we confirmed that a millimeter‐size motor, which was prepared from the obtained film, rotates at 15,160 rpm under a no‐load test. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 159(2): 1–6, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20215     

A high‐speed‐execution method for a program written in factory‐automation‐cell control language

A new programming method called Cell Control Language (CCL) for factory‐automation systems has been developed. In CCL programming, an action sequence of each functional unit in each operation mode is first described as a net module. This net module is a subclass of Petri nets, and this subclass is equivalent to a state‐transition diagram (a conventional design method). Several net modules are merged and translated into an internal code program, which is executed in a cell controller. To improve the ability of the controller to respond to state changes, we developed two new methods based on CCL. One is that for efficiently describing a CCL program, which needs a quick response to state changes, and the other is that for executing the program in a short cycle. The execution method was applied to an actual cell controller, which confirmed that it can improve the ability to respond to state changes without reducing the total control performance. © 2003 Wiley Periodicals, Inc. Electr Eng Jpn, 144(2): 34–42, 2003; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.10167     

Stochastic H∞ control problem with state‐dependent noise for weakly coupled large‐scale systems

In this paper, stochastic H_∞ state feedback control with state‐dependent noise for weakly coupled large‐scale systems is discussed. After establishing the asymptotic structure of the stochastic algebraic Riccati equation (SARE), a new iterative algorithm that combines the Newton's method with the fixed‐point algorithm is derived for the first time. As a result, both the quadratic convergence and the reduced‐order computation in the same dimension of the subsystems are attained. Copyright © 2007 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.     

A force‐reflecting friction‐free bilateral system based on a twin drive control system with torsional vibration suppression

This paper proposes a new friction‐free bilateral system based on twin drive control system considering resonant frequency for bilateral systems. The twin drive system consists of two motors which are coupled by the differential gear. The output torque becomes a different torque of both motors. The nonlinear friction torque of the twin drive system can be easily compensated. However, the resonant frequency and the antiresonant frequency caused by the tensional vibration exist. This paper proposes a new two‐mass model of twin drive system that supresses torsional vibration by state feedback. The proposed control method is applied to the bilateral robot, and the effectiveness of the control method is confirmed by the experimental results. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 159(1): 72–79, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20355     

Model‐free recursive LQ controller design (learning LQ control)

A new data‐based iterative self‐optimizing approach to practical design (learning/adaptive process) of the infinite‐horizon LQ regulator is proposed. Optimality is given by a certain orthogonality condition of response signals, and the global convergence of feedback gain is proved for MIMO systems by an expansion of the Riccati equation. The design is applied to stabilizing control and steady state error‐less control of physical systems. Copyright © 2004 John Wiley & Sons, Ltd.     

A simple instant‐estimation method for time‐average quantities of single‐phase power and application to single‐phase power grid connection by inverter

This paper presents and analyzes a new simple instant‐estimation method for time‐average quantities such as rms values of voltage and current, active and reactive powers, and power factor for single‐phase power with the fundamental component of constant or nearly constant frequency by measuring instantaneous values of voltage and current. According to the analyses, the method can instantly estimate time‐average values with accuracy of the fundamental frequency, and estimation accuracy of the power factor is about two times better than that of voltage, current, and powers. The instant‐estimation method is simple and can be easily applied to single‐phase power control systems that are expected to control instantly and continuously power factor on a single‐phase grid by inverter. Based on the proposed instant‐estimation method, two methods for such power control systems are also proposed and their usefulness is verified through simulations. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 159(2): 34–43, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20236     

Hierarchical multi‐label classification based on over‐sampling and hierarchy constraint for gene function prediction

Hierarchical multi‐label classification (HMC) is a variant of classification where instances may belong to multiple classes at the same time and these classes are organized in a hierarchy. Gene function prediction is a complicated HMC problem with large class number and usually strongly imbalanced class distributions. This paper proposes an improved HMC method based on over‐sampling and hierarchy constraint for solving the gene function prediction problem. The HMC task is transferred into a set of binary support vector machine (SVM) classification tasks. Then, two measures are implemented to enhance the HMC performance by introducing the hierarchy constraint into learning procedures. Firstly, for imbalanced classes, a hierarchical synthetic minority over‐sampling technique (SMOTE) is proposed as over‐sampling preprocessing to improve the SVM learning performance. Secondly, an improved True Path Rule (TPR) ensemble approach is introduced to combine the results of binary probabilistic SVM classifications. It can improve the classification results and guarantee the hierarchy constraint of classes. Experiment results on four benchmark FunCat Yeast datasets show that the proposed method significantly outperforms the basic TPR method and the Flat ensemble method. © 2012 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.     

Iterative learning control for a class of non‐affine‐in‐input processes in Hilbert space

In this paper, iterative learning control (ILC) of a class of non‐affine‐in‐input processes is considered in Hilbert space, where the plant operators are quite general in the sense that they could be static or dynamic, differentiable or non‐differentiable, continuous‐time or discrete‐time, and so forth. The control problem is first transformed to a problem of solving global implicit function to ensure the uniqueness of desired control input. Then, two contraction mapping‐based ILC schemes are proposed in terms of the continuous differentiability of process model, where the learning convergence condition is derived through rigorous analysis. The proposed ILC schemes make full use of the process repetition, deal with system uncertainties easily, and are effective to infinite‐dimensional or distributed parameter systems. In the end, the learning controller is applied to the boundary output control of a class of anaerobic digestion process for wastewater treatment. The control efficacy is verified by simulation. Copyright © 2013 John Wiley & Sons, Ltd.     

Performance evaluation of 6000‐A‐class superconducting field windings for 600‐MW‐class superconducting generators

Superconducting generators have many advantages such as increasing generator efficiency and improving power system stability. In Japan, a national project has been conducted since 2000 which is aimed at the development of fundamental technologies required for high‐output‐density and large‐capacity superconducting generators. This paper describes the results of this project, focusing on 6000‐A‐class field winding development. Copyright © 2004 Wiley Periodicals, Inc. A superconducting generator with a high output density and a large capacity has inherent factors that decrease superconducting stability. These are: (1) increase in the magnetic field in the winding which is caused by the increase in winding current density and (2) difficulty in fabricating windings which increases as a conductor diameter becomes larger. To secure the stability, we adopted a higher‐copper‐content conductor and a design that increases winding fixing pressure, along with devising a winding method that accommodates larger conductor diameter. These improvements were applied to a partial model of a 600‐MW field winding. Test results of the model showed good stability, indicating that design and fabrication technique for a 6000‐A‐class superconducting field winding has been successfully evaluated. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 159(2): 7– 18, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20332     

Online concurrent reinforcement learning algorithm to solve two‐player zero‐sum games for partially unknown nonlinear continuous‐time systems

Online adaptive optimal control methods based on reinforcement learning algorithms typically need to check for the persistence of excitation condition, which is necessary to be known a priori for convergence of the algorithm. However, this condition is often infeasible to implement or monitor online. This paper proposes an online concurrent reinforcement learning algorithm (CRLA) based on neural networks (NNs) to solve the H _∞ control problem of partially unknown continuous‐time systems, in which the need for persistence of excitation condition is relaxed by using the idea of concurrent learning. First, H _∞ control problem is formulated as a two‐player zero‐sum game, and then, online CRLA is employed to obtain the approximation of the optimal value and the Nash equilibrium of the game. The proposed algorithm is implemented on actor–critic–disturbance NN approximator structure to obtain the solution of the Hamilton–Jacobi–Isaacs equation online forward in time. During the implementation of the algorithm, the control input that acts as one player attempts to make the optimal control while the other player, that is, disturbance, tries to make the worst‐case possible disturbance. Novel update laws are derived for adaptation of the critic and actor NN weights. The stability of the closed‐loop system is guaranteed using Lyapunov technique, and the convergence to the Nash solution of the game is obtained. Simulation results show the effectiveness of the proposed method. Copyright © 2014 John Wiley & Sons, Ltd.     

Rule pool updating through Sarsa‐learning to improve adaptability in changing environments

Genetic network programming (GNP) is a new evolutionary algorithm using the directed graph as its chromosome. A GNP‐based rule accumulation (GNP‐RA) method was proposed previously for multiagent control. However, in changing environments where new situations appear frequently, the old rules in the rule pool become incompetent for guiding the agent's actions, and therefore updating them becomes necessary. This paper proposes a more robust rule‐based model which can adapt to the environment changes. In order to realize this, Sarsa‐learning is used as a tool to update the rules to cope with the unexperienced situations in new environments. Furthermore, Sarsa‐learning helps to generate better rules by selecting really important judgments and actions during training. In addition, the ε‐greedy policy of Sarsa enables GNP‐RA to explore the solutions space sufficiently, generating more rules. Simulations on the tile world problem show that the proposed method outperforms the previous ones, namely GP and reinforcement learning. © 2014 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.     

Broadband characteristics analysis of semicircle‐type bow‐tie antenna with hole slots

For electromagnetic compatibility, broadband antennas are important for measurements of fast pulse transient electromagnetic phenomena and broadband characteristics due to noise and high‐frequency interference. We analyzed the characteristics of a semicircle type bow‐tie antenna with various slots using the FDTD method. It was shown from the simulation results that the shape and position of the slot influenced greatly the broadband characteristics of the antenna. We confirmed that a semicircle type bow‐tie antenna with a triangle slot was effective for a broadband antenna. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 159(4): 47–53, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20252     

Learning of real robot's inverse dynamics by a forward‐propagation learning rule

A forward‐propagation learning rule (FPL) has been proposed for a neural network (NN) to learn an inverse model of a controlled object. A feature of FPL is that the trajectory error propagates forward in NN and appropriate values of two learning parameters are required to be set. FPL has only been simulated to several kinds of controlled objects such as a two‐link arm in a horizontal plane. In this work, we applied FPL to AIBO and showed the validity of FPL on a real controlled object. At first, we tested a learning experiment of an inverse dynamic of a two‐link arm in a sagittal plane with viscosity and Coulomb friction by computer simulation. In this simulation, a low‐pass filter (LPF) was applied to realized trajectories because coulomb friction vibrates them. From the simulation results, we found that the learning process is stable by some adequate sets of the learning parameters although it is more sensitive to the values of the parameters owing to friction and gravity terms. Finally, we tested applying FPL to motor control of AIBO's leg. The inverse dynamics model was acquired by FPL with only about 150 learning iterations. From these results, the validity of the FPL was confirmed by the real robot control experiments. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 161(4): 38–48, 2007; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20456     

Parameter design of event‐driven logic controller for two‐wheeled vehicle

Programmable Logic Controllers (PLCs) are widely used in industry. In PLC‐based control systems, low‐resolution (especially ON‐/OFF) sensors are inexpensive, and actuators are commonly used because they are compatible with programming languages used in PLCs. PLC switches the actuators ON/OFF as the state of the sensor changes between ON/OFF. In designing PLC‐based systems, the design of the parameters of these sensors and actuators (e.g., position of limit switches and torque of motors) is an important problem because they affect the overall performance of the system. This problem, however, has not yet been fully discussed. In the present paper, a systematic design method for this problem is developed. The main concept is to express the model of the system as a Mixed Logical Dynamical System (MLDS) and to formulate the problem as a mathematical programming problem. The developed idea is applied to the line‐following control of a two‐wheeled vehicle. The usefulness of the proposed method is demonstrated through simulation and experiments. © 2007 Wiley Periodicals, Inc. Electr Eng Jpn, 162(2): 51–60, 2008; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/eej.20584