Partially Observable Markov Decision Process Approximations for Adaptive Sensing

Adaptive sensing involves actively managing sensor resources to achieve a sensing task, such as object detection, classification, and tracking, and represents a promising direction for new applications of discrete event system methods. We describe an approach to adaptive sensing based on approximately solving a partially observable Markov decision process (POMDP) formulation of the problem. Such approximations are necessary because of the very large state space involved in practical adaptive sensing problems, precluding exact computation of optimal solutions. We review the theory of POMDPs and show how the theory applies to adaptive sensing problems. We then describe a variety of approximation methods, with examples to illustrate their application in adaptive sensing. The examples also demonstrate the gains that are possible from nonmyopic methods relative to myopic methods, and highlight some insights into the dependence of such gains on the sensing resources and environment.

Integrated maintenance and operations decision making with imperfect degradation state observations

In highly flexible and integrated manufacturing systems, such as semiconductor fabs, strong interactions between the equipment condition, operations executed on the various machines and the outgoing product quality necessitate integrated decision making in the domains of maintenance scheduling and production operations. Furthermore, in highly complex manufacturing equipment, the underlying condition is not directly observable and can only be inferred probabilistically from the available sensor readings. In order to deal with interactions between maintenance and production operations in Flexible Manufacturing Systems (FMSs) in which equipment conditions are not perfectly observable, we propose in this paper a decision-making method based on a Partially Observable Markov Decision Processes (POMDP's), yielding an integrated policy in the realms of maintenance scheduling and production sequencing. Optimization was pursued using a metaheuristic method that used the results of discrete-event simulations of the underlying manufacturing system. The new approach is demonstrated in simulations of a generic semiconductor manufacturing cluster tool. The results showed that, regardless of uncertainties in the knowledge of actual equipment conditions, jointly making maintenance and production sequencing decisions consistently outperforms the current practice of making these decisions separately.     

The Hidden Information State model: A practical framework for POMDP-based spoken dialogue management

This paper explains how Partially Observable Markov Decision Processes (POMDPs) can provide a principled mathematical framework for modelling the inherent uncertainty in spoken dialogue systems. It briefly summarises the basic mathematics and explains why exact optimisation is intractable. It then describes in some detail a form of approximation called the Hidden Information State model which does scale and which can be used to build practical systems. A prototype HIS system for the tourist information domain is evaluated and compared with a baseline MDP system using both user simulations and a live user trial. The results give strong support to the central contention that the POMDP-based framework is both a tractable and powerful approach to building more robust spoken dialogue systems.     

基于杂合标准的POMDP值迭代求解算法*

基于点的值迭代方法是求解部分可观测马尔科夫决策过程(POMDP)问题的一类有效算法.目前基于点的值迭代算法大都基于单一启发式标准探索信念点集,从而限制算法效果.基于此种情况,文中提出基于杂合标准探索信念点集的值迭代算法(HHVI),可以同时维持值函数的上界和下界.在扩展探索点集时,选取值函数上下界差值大于阈值的信念点进行扩展,并且在值函数上下界差值大于阈值的后继信念点中选择与已探索点集距离最远的信念点进行探索,保证探索点集尽量有效分布于可达信念空间内.在4个基准问题上的实验表明,HHVI能保证收敛效率,并能收敛到更好的全局最优解.     

基于部分可观测Markov决策过程理论的盾构推进载荷规划

针对盾构掘进过程中位姿控制问题,提出了基于部分可观测马尔科夫决策过程(Partially observable Markov decision processes, POMDP)理论的推进载荷规划方法。在推进载荷规划模型中,将盾构自动纠偏看成不确定环境下序贯决策问题,充分考虑掘进过程中随机因素的影响,将盾构掘进过程中受到的阻力、推进载荷和盾构位姿分别定义为POMDP的状态集、行动集和观测集,然后重点讨论了信念状态、状态转移函数和观测函数等几个关键参数的获取方法。在计算值函数时,考虑了盾构位姿偏离程度和盾构载荷平稳程度对推进载荷决策的影响,建立了立即收益函数和长期折扣收益函数,并采用基于点的值迭代算法寻求推进载荷最优规划策略。针对天津地铁9号线进行了案例分析,结果表明基于POMDP的推进载荷规划方法是合理有效的,能够顺应掘进阻力随机变化。     

功率受限和自动重传请求条件下系统吞吐量最大化的研究

本文研究了具有ARQ功能的基于衰落信道和数据链路层缓冲区队列状态的资源最优分配问题，目标是通过自适应调整功率分配和调制方式，在系统平均功率的限制下，使系统的吞吐量达到最大。在这个系统中并不限制ARQ的重发次数，所以最大化系统的吞吐量等效于使链路层的缓冲区溢出的数据包最小。本文把这样一个优化问题构造为马尔可夫决策过程，并提出了用动态规划解决该问题的方法。出于实用性的考虑，本文还提出了一种简单的次优资源分配方法，仿真结果显示这种方法与最优的调度方法性能非常接近。     

基于实例的POMDP问题的近似求解

结合启发式求解和增强学习技术,深入研究了基于实例的POMDP问题的近似求解算法,包括基于最近邻算法法的NNI及它的参数化增强版本ENNI和基于局部加权回归算法的LWI,并通过实验对比,给出了相应算法在实际应用中的性能。实验证明,基于实例的方法来求解POMDP问题,能够获得性能较好的次优解。     

一种动态不确定性环境中的持续规划系统

规划是人工智能研究的一个重要方向,具有极其广泛的应用背景．近年来,研究重点已经转移到动态不确定性环境中的规划问题．该文将部分可观察马尔可夫决策过程(POMDP)和过程性推理系统(PRS)的优点相结合,提出一种对动态不确定环境具有更全面适应能力的持续规划系统——POMDPRS．该系统利用PRS的持续规划机制,交叉地进行规划与执行,在一定条件下提高了动态环境中POMDP决策的效率;另一方面,用POMDP的概率分布信念模型和极大效用原理替代PRS的一阶逻辑信念表示和计划选择机制,大大增强了处理环境不确定性的能力．     

基于吞吐量最大化的自适应传输控制

研究了具有ARQ功能的基于衰落信道和数据链路层缓冲区队列状态的资源最优分配问题,为了通过自适应调整功率分配和调制方式,在系统平均功率的限制下,使系统的吞吐量达到最大,该文把这个优化问题构造为马尔可夫决策过程,并提出了用动态规划解决该问题的方法。     

基于循环卷积神经网络的POMDP值迭代算法

基于卷积神经网络的部分可观测马尔科夫决策过程(POMDP)值迭代算法QMDP-net在无先验知识的情况下具有较好的性能表现,但其存在训练效果不稳定、参数敏感等优化难题.提出基于循环卷积神经网络的POMDP值迭代算法RQMDP-net,使用门控循环单元网络实现值迭代更新,在保留输入和递归权重矩阵卷积特性的同时增强网络时序...