Control allocation-based adaptive control for greenhouse climate

This paper presents an adaptive approach to greenhouse climate control, as part of an integrated control and management system for greenhouse production. In this approach, an adaptive control algorithm is first derived to guarantee the asymptotic convergence of the closed system with uncertainty, then using that control algorithm, a controller is designed to satisfy the demands for heat and mass fluxes to maintain inside temperature, humidity and CO₂ concentration at their desired values. Instead of applying the original adaptive control inputs directly, second, a control allocation technique is applied to distribute the demands of the heat and mass fluxes to the actuators by minimising tracking errors and energy consumption. To find an energy-saving solution, both single-objective optimisation (SOO) and multiobjective optimisation (MOO) in the control allocation structure are considered. The advantage of the proposed approach is that it does not require any a priori knowledge of the uncertainty bounds, and the simulation results illustrate the effectiveness of the proposed control scheme. It also indicates that MOO saves more energy in the control process.     

Nonlinear MPC based on a Volterra series model for greenhouse temperature control using natural ventilation

Suitable environmental conditions are a fundamental issue in greenhouse crop growth and can be achieved by advanced climate control strategies. In different climatic zones, natural ventilation is used to regulate both the greenhouse temperature and humidity. In mild climates, the greatest problem faced by far in greenhouse climate control is cooling, which, for dynamical reasons, leads to natural ventilation as a standard tool. This work addresses the design of a nonlinear model predictive control (NMPC) strategy for greenhouse temperature control using natural ventilation. The NMPC strategy is based on a second-order Volterra series model identified from experimental input/output data of a greenhouse. These models, representing the simple and logical extension of convolution models, can be used to approximate the nonlinear dynamic effect of the ventilation and other environmental conditions on the greenhouse temperature. The developed NMPC is applied to a greenhouse and the control performance of the proposed strategy will be illustrated by means of experimental results.     

物联网技术在高效农业种植中的应用研究

我国是一个农业大国,以温室大棚为主体的高效农业种植是目前非常关注的问题。论文设计了一种基于物联网技术的温室智能监控系统,该系统采用n RF2401组建无线传感网络,实现对温室内多个环境因子的实时监测与控制。经测试,该系统运行稳定,达到智能监控的效果。     

基于局部合作的RoboCup多智能体Q-学习

针对多智能体Q-学习中存在的联合动作指数级增长问题,采用-种局部合作的Q-学习方法,在智能体之间有协作时才考察联合动作,否则只进行简单的个体智能体的Q-学习,从而减少学习时所要考察的状态-动作对值。在机器人足球仿真2D平台上进行的实验表明,该方法比常用多智能体强化学习技术具有更高的效率。     

Greenhouse climate fuzzy adaptive control considering energy saving

This paper proposes a fuzzy adaptive control approach to solve greenhouse climate control problem. The aim is to ensure the controlled environmental variables to track their desired trajectories so as to create a favorable environment for crop growth. In this method, a feedback linearization technique is first introduced to derive the control laws of heating, fogging and CO2 injection, then to compensate for the saturation of the actuators, a fuzzy logic system (FLS) is used to approximate the differences between controller outputs and actuator outputs due to actuator constraints. A robust control term is introduced to eliminate the impact of external disturbances and model uncertainty, and finally, Lyapunov stability analysis is performed to guarantee the convergence of the closed-loop system. Taking into account the fact that the crop is usually insensitive to the change of the environment inside the greenhouse during a short time interval, a certain amount of tracking error of the environmental variables is usually acceptable, which means that the environmental variables need only be driven into the corresponding target intervals. In this case, an energy-saving management mechanism is designed to reduce the energy consumption as much as possible. The simulation results illustrate the effectiveness of the proposed control scheme.

     

Model-Free control performance improvement using virtual reference feedback tuning and reinforcement Q-learning

This paper proposes the combination of two model-free controller tuning techniques, namely linear virtual reference feedback tuning (VRFT) and nonlinear state-feedback Q-learning, referred to as a new mixed VRFT-Q learning approach. VRFT is first used to find stabilising feedback controller using input-output experimental data from the process in a model reference tracking setting. Reinforcement Q-learning is next applied in the same setting using input-state experimental data collected under perturbed VRFT to ensure good exploration. The Q-learning controller learned with a batch fitted Q iteration algorithm uses two neural networks, one for the Q-function estimator and one for the controller, respectively. The VRFT-Q learning approach is validated on position control of a two-degrees-of-motion open-loop stable multi input-multi output (MIMO) aerodynamic system (AS). Extensive simulations for the two independent control channels of the MIMO AS show that the Q-learning controllers clearly improve performance over the VRFT controllers.     

基于多智能体增强学习的公交驻站控制方法

车辆驻站是减少串车现象和改善公交服务可靠性的常用且有效控制策略,其执行过程需要在随机交互的系统环境中进行动态决策。考虑实时公交运营信息的可获得性,研究智能体完全合作环境下公交车辆驻站增强学习控制问题,建立基于多智能体系统的单线公交控制概念模型,描述学习框架下包括智能体状态、动作集、收益函数、协调机制等主要元素,采用hysteretic Q-learning算法求解问题。仿真实验结果表明该方法能有效防止串车现象并保持单线公交服务系统车头时距的均衡性。     

基于Q学习的单路口交通信号协调控制

Q学习通过与外部环境的交互来进行单路口的交通信号自适应控制。在城市交通愈加拥堵的时代背景下,为了缓解交通拥堵,提出一种结合SCOOT系统对绿信比优化方法的Q学习算法。本文将SCOOT系统中对绿信比优化的方法与Q学习相结合,即通过结合车均延误率以及停车次数等时间因素以及经济因素2方面,建立新的数学模型来作为本算法的成本函数并建立一种连续的奖惩函数,在此基础上详细介绍Q学习算法在单路口上的运行过程并且通过与Webster延误率和基于最小车均延误率的Q学习进行横向对比,验证了此算法优于定时控制以及基于车均延误的Q学习算法。相对于这2种算法,本文提出的算法更加适合单路口的绿信比优化。     

Q-learning of complex behaviours on a six-legged walking machine

We present work on a six-legged walking machine that uses a hierarchical version of [C.J.C.H. Watkins, Learning with delayed rewards, Ph.D. Thesis, Psychology Department, Cambridge University, 1989] Q-learning (HQL) to learn both: the elementary swing and stance movements of individual legs as well as the overall coordination scheme to perform forward movements. The architecture consists of a hierarchy of local controllers implemented in layers. The lowest layer consists of control modules performing elementary actions, like moving a leg up, down, left or right to achieve the elementary swing and stance motions for individual legs. The next level consists of controllers that learn to perform more complex tasks like forward movement by using the previously learned, lower level modules. The work is related to similar, although simulation based, work [L.J. Lin, Reinforcement learning for robots using neural networks, Ph.D. Thesis, Carnegie Mellon University, 1993] on hierarchical reinforcement-learning and [S.P. Singh, learning to solve Markovian decision problems, Ph.D. Thesis, Department of Computer Science at the University of Massachusetts, 1994] on compositional Q-learning. We report on the HQL architecture as well as on its implementation on the walking machine Sir Arthur. Results from experiments carried out on the real robot are reported to show the applicability of the HQL approach to real world robot problems.     

基于Q-learning的离散时间多智能体系统一致性

针对模型未知的一类离散时间多智能体系统,本文提出了一种Q-learning方法实现多智能体系统的一致性控制.该方法不依赖于系统模型,能够利用系统数据迭代求解出可使给定目标函数最小的控制律,使所有智能体的状态实现一致.通过各个智能体所产生的系统数据,采用策略迭代的方法实时更新求解得到多智能体系统的控制律,并对所提Q-le...     

NSGA-II-based nonlinear PID controller tuning of greenhouse climate for reducing costs and improving performances

The traditional tuning scheme of proportional, integral, and derivative (PID) controller parameters usually lay more emphasis on control performances than economic profits. As a result, the corresponding control performance is improved, but such case may lead to high production costs. In this paper, a new tuning methodology for multiple PID controllers from an economic point of view by incorporating multiple performance measures and production costs based on nondominated sorting genetic algorithm-II (NSGA-II) is presented. A model of nonlinear thermodynamic laws between numerous system variables affecting the greenhouse climate is formulated. The proposed tuning scheme is tested through step responses for greenhouse climate control by minimizing the indices of overall performance and production cost in a simulation experiment. The results show that the controllers by tuning the gain parameters can achieve good control performance at a relatively low cost. Maybe it is a quite effective and promising tuning method by using this method in the complex greenhouse production.     

A long short-term memory-based model for greenhouse climate prediction

Greenhouses can grow many off-season vegetables and fruits, which improves people's quality of life. Greenhouses can also help crops resist natural disasters and ensure the stable growth of crops. However, it is highly challenging to carefully control the greenhouse climate. Therefore, the proposal of a greenhouse climate prediction model provides a way to solve this challenge. We focus on the six climatic factors that affect crops growth, including temperature, humidity, illumination, carbon dioxide concentration, soil temperature and soil humidity, and propose a GCP_lstm model for greenhouse climate prediction. The climate change in greenhouse is nonlinear, so we use long short-term memory (LSTM) model to capture the dependence between historical climate data. Moreover, the short-term climate has a greater impact on the future trend of greenhouse climate change. Therefore, we added a 5-min time sliding window through the analysis experiment. In addition, sensors sometimes collect wrong climate data. Based on the existence of abnormal data, our model still has good robustness. We experienced our method on the data sets of three vegetables: tomato, cucumber and pepper. The comparison shows that our method is better than other comparison models.     

温室环境测控系统的设计与运行

为满足温室作物对生长环境的要求,达到作物生长根上、根下环境因子检测与控制的高度集成,利用已有的作物根上环境因子CAN现场总线控制系统,设计了CAN总线的接口电路、根下营养液组分的信号采集电路、根下营养液组分控制系统.通过对营养液施加一个自适应控制量,将动态矩阵控制算法应用到营养液组分控制中;并将营养液组分控制软件集成到CAN现场总线中,实现了根上、根下环境因子的一体化控制.该算法建模简单,便于工程应用.现场运行结果表明,其控制效果良好.     

Distributed Reinforcement Learning Control for Batch Sequencing and Sizing in Just-In-Time Manufacturing Systems

This paper presents an approach that is suitable for Just-In-Time (JIT) production for multi-objective scheduling problem in dynamically changing shop floor environment. The proposed distributed learning and control (DLC) approach integrates part-driven distributed arrival time control (DATC) and machine-driven distributed reinforcement learning based control. With DATC, part controllers adjust their associated parts' arrival time to minimize due-date deviation. Within the restricted pattern of arrivals, machine controllers are concurrently searching for optimal dispatching policies. The machine control problem is modeled as Semi Markov Decision Process (SMDP) and solved using Q-learning. The DLC algorithms are evaluated using simulation for two types of manufacturing systems: family scheduling and dynamic batch sizing. Results show that DLC algorithms achieve significant performance improvement over usual dispatching rules in complex real-time shop floor control problems for JIT production.     

基于非策略Q-学习的网络控制系统最优跟踪控制

针对具有数据包丢失的网络化控制系统跟踪控制问题,提出一种非策略Q-学习方法,完全利用可测数据,在系统模型参数未知并且网络通信存在数据丢失的情况下,实现系统以近似最优的方式跟踪目标.首先,刻画具有数据包丢失的网络控制系统,提出线性离散网络控制系统跟踪控制问题;然后,设计一个Smith预测器补偿数据包丢失对网络控制系统性能的影响,构建具有数据包丢失补偿的网络控制系统最优跟踪控制问题;最后,融合动态规划和强化学习方法,提出一种非策略Q-学习算法.算法的优点是:不要求系统模型参数已知,利用网络控制系统可测数据,学习基于预测器状态反馈的最优跟踪控制策略;并且该算法能够保证基于Q-函数的迭代Bellman方程解的无偏性.通过仿真验证所提方法的有效性.     

The significance of crop co-states for receding horizon optimal control of greenhouse climate

While a tomato crop grows on the time-scale of weeks, the greenhouse climate changes on a time-scale of minutes. The economic optimal control problem of producing good quality crops against minimum input of resources is tackled by a two time-scale decomposition. First, the sub-problem associated to the slow crop evolution is solved off-line, leading to a seasonal pattern for the co-states of the amount of assimilates produced by photosynthesis, and the fruit and leaf weights. These co-states can be interpreted as the marginal prices of a unit of assimilate, leaf and fruit. Next, they are used in the goal function of an on-line receding horizon control (RHOC) of the greenhouse climate, thus balancing costs of heating and CO₂-dosage against predicted benefits from harvesting, while profiting as much as possible from the available solar radiation. Simulations using the time-varying co-states are compared to experimental results obtained with fixed co-states. It appears that the on-line control is sensitive to the time evolution of the co-states, suggesting that it is advantageous to repeat the seasonal optimisation from time to time to adjust the co-states to the past weather and realised crop state.     

引入势场及陷阱搜索的强化学习路径规划算法

移动机器人在复杂环境中移动难以得到较优的路径,基于马尔可夫过程的Q学习（Q-learning）算法能通过试错学习取得较优的路径,但这种方法收敛速度慢,迭代次数多,且试错方式无法应用于真实的环境中。在Q-learning算法中加入引力势场作为初始环境先验信息,在其基础上对环境进行陷阱区域逐层搜索,剔除凹形陷阱区域[Q]值迭代,加快了路径规划的收敛速度。同时取消对障碍物的试错学习,使算法在初始状态就能有效避开障碍物,适用于真实环境中直接学习。利用python及pygame模块建立复杂地图,验证加入初始引力势场和陷阱搜索的改进Q-learning算法路径规划效果。仿真实验表明,改进算法能在较少的迭代次数后,快速有效地到达目标位置,且路径较优。     

基于Q学习的无线传感器网络自组织方法研究

为实现无线传感器网络中数据传输效率与能量节省的综合性能优化,提出一种基于Q学习的自组织协议方法,将无线传感器网络的每个节点映射为一个Agent,通过学习训练,使得每个Agent可以选择一个较优的转发方向,从而实现无线传感器网络的自组织。实例分析表明,应用Q学习构建的自组织传感器网络能够提高数据传输的能量效率,延长网络生存期。     

IntelliGrow: a greenhouse component-based climate control system

A new greenhouse climate control system has been constructed with the objective of decreasing energy consumption while maintaining, or even increasing, plant production. The system is based on the use of mathematical models for estimating the absorption of irradiance, leaf photosynthesis and respiration. The model builds on a general leaf model that with a few modifications can be used for different greenhouse crops. The temperature, which was controlled according to the natural irradiance, was allowed to vary considerably more than in a standard climate. Under low light conditions, energy use was reduced because the temperature was lowered. In contrast, when irradiance is higher, the plants seem able to utilize both a higher temperature and CO₂. During nighttimes the temperature was lower than in standard climate. The thermal screens were used according to a screen simulation system. The system balances the energy costs saved via isolation against the production loss caused by the decrease in irradiance. A six-month trial of the system resulted in energy savings ranging from 8% in late spring (April–June) to 40% in early spring (March–May). During the winters 1997–98 and 1998–99 the accumulated energy savings per season varied between 20% and 38% at two different locations in Denmark.     

Backward Q-learning: The combination of Sarsa algorithm and Q-learning

Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning and the Sarsa algorithms, but they possess different characteristics. Generally speaking, the Sarsa algorithm has faster convergence characteristics, while the Q-learning algorithm has a better final performance. However, Sarsa algorithm is easily stuck in the local minimum and Q-learning needs longer time to learn. Most literatures investigated the action selection policy. Instead of studying an action selection strategy, this paper focuses on how to combine Q-learning with the Sarsa algorithm, and presents a new method, called backward Q-learning, which can be implemented in the Sarsa algorithm and Q-learning. The backward Q-learning algorithm directly tunes the Q-values, and then the Q-values will indirectly affect the action selection policy. Therefore, the proposed RL algorithms can enhance learning speed and improve final performance. Finally, three experimental results including cliff walk, mountain car, and cart–pole balancing control system are utilized to verify the feasibility and effectiveness of the proposed scheme. All the simulations illustrate that the backward Q-learning based RL algorithm outperforms the well-known Q-learning and the Sarsa algorithm.