马尔可夫决策过程两种抽象模式

抽象层次上马尔可夫决策过程的引入,使得人们可简洁地、陈述地表达复杂的马尔可夫决策过程,解决常规马尔可夫决策过程(MDPs)在实际中所遇到的大型状态空间的表达问题.介绍了结构型和概括型两种不同类型抽象马尔可夫决策过程基本概念以及在各种典型抽象MDPs中的最优策略的精确或近似算法,其中包括与常规MDPs根本不同的一个算法:把Bellman方程推广到抽象状态空间的方法,并且对它们的研究历史进行总结和对它们的发展做一些展望,使得人们对它们有一个透彻的、全面而又重点的理解.     

融合模糊逻辑与马尔可夫决策过程的边缘计算切换策略

在边缘计算切换策略中,针对马尔可夫决策过程（Markov decision process,MDP）传输时延高且环境适应能力差等问题,提出了一种融合模糊逻辑与马尔可夫决策过程的边缘计算切换策略。采用模糊逻辑算法将系统参数模糊化,并且将模糊值引入适应度函数,保证系统参数能够有效融合;利用差分进化算法求解适应度函数最大值,从而选取出该环境的最优规则,提高边缘计算对环境的适应能力;将适应度函数引入MDP,提高系统综合性能。该方案将移动智能设备作为任务卸载发起方,将边缘服务器作为任务卸载对象,对一维MDP切换策略、一维仅时延MDP切换策略、二维MDP切换策略、模糊逻辑MDP切换策略、最小距离切换算法和最小时延切换算法进行仿真。仿真结果表明,模糊逻辑MDP的边缘计算切换策略的任务执行平均时长为608.8 s,较一维MDP切换策略、一维仅时延MDP切换策略、二维MDP切换策略、最小距离切换算法和最小时延切换算法分别降低了27.2%、8.6%、37.1%、41%和22.3%。该方案在提高了基于MDP的边缘计算切换策略的环境适应性的同时,大幅降低了边缘计算的传输时延。     

马尔可夫决策过程复杂性的熵测度

应用Shannon熵和其他熵指数来度量马尔可夫决策的复杂性．将马尔可夫链的复杂性、不确定性和不可预测性的度量扩展到马尔可夫决策，提出一套基于信息理论的复杂性度量方法，可用于随机和确定性策略下的完全观测和不完全观测马尔可夫决策．对有关数值进行仿真研究，并给出了计算结果．     

马尔可夫决策过程自适应决策的进展

在介绍一般马尔可夫决策过程的基础上,分析了当前主要马尔可夫过程自适应决策方法的基本思想、具体算法实现以及相应结论,总结了现有马尔可夫过程自适应决策算法的特点,并指出了需要进一步解决的问题。     

双马尔可夫决策过程联合模型

人类在处理问题中往往分为两个层次,首先在整体上把握问题,即提出大体方案,然后再具体实施.也就是说人类就是具有多分辨率智能系统的极好例子,他能够在多个层次上从底向上泛化(即看问题角度粒度变"粗",它类似于抽象),并且又能从顶向下进行实例化(即看问题角度变"细",它类似于具体化).由此构造了由在双层(理想空间即泛化和实际空间即实例化)上各自运行的马尔可夫决策过程组成的半马尔可夫决策过程,称之为双马尔可夫决策过程联合模型.然后讨论该联合模型的最优策略算法,最后给出一个实例说明双马尔可夫决策联合模型能够经济地节约"思想",是运算有效性和可行性的一个很好的折中.     

关系马尔可夫网综述

统计关系学习是人工智能领域一个新的研究方向。它通过概率推理模型与逻辑的结合,或概率推理模型与关系模式的结合,来达到更高的预测或分类的准确度。它在机器学习和数据挖掘领域具有广泛的应用前景。详细介绍了一种重要的统计关系模型——关系马尔可夫网的理论模型,并总结关系马尔可夫网当前的研究现状,分析了关系马尔可夫网目前存在的问题以及未来的研究方向。     

基于马尔可夫决策过程的群体动画运动轨迹生成

近些年来,群体动画在机器人学、电影、游戏等领域得到了广泛的研究和应用,但传统的群体动画技术均涉及复杂的运动规划或碰撞避免操作,计算效率较低.本文提出了一种基于马尔可夫决策过程（MDPs）的群体动画运动轨迹生成算法,该算法无需碰撞检测即可生成各智能体的无碰撞运动轨迹.同时本文还提出了一种改进的值迭代算法用于求解马尔可夫决策过程的状态-值,利用该算法在栅格环境中进行实验,结果表明该算法的计算效率明显高于使用欧氏距离作为启发式的值迭代算法和Dijkstra算法.利用本文提出的运动轨迹生成算法在三维（3D）动画场景中进行群体动画仿真实验,结果表明该算法可实现群体无碰撞地朝向目标运动,并具有多样性.     

基于马尔可夫决策过程的无线传感器网络速率控制

无线传感器网络近年来得到了较为广泛的应用,其中能耗问题为该领域的研究热点问题。同时,随着无线传感器网络技术的不断发展,现在在传感器网络中常使用多速率进行网络传输,此多速率的属性提供了可进一步提高网络能耗性能的机会。本文提出一种基于马尔可夫决策过程控制无线传感器网络的多速率之间的转换,进而达到使网络更加节能的目的。仿真结果表明,在不影响通信质量的情况下,网络能耗性能得到了提高。     

行动驱动的马尔可夫决策过程及在RoboCup中的应用

为了更好地解决一类特殊的Agent决策问题,提出行动驱动的马尔可夫决策过程的概念并分析了其理论模型.另外,提出行动驱动的马尔可夫决策过程相关问题的求解算法,并在RoboCup仿真2D比赛的不离身带球问题中对算法进行了实验.实验结果表明,新算法使Agent的带球性能有了较大的提高.新算法已经用于中国科大蓝鹰仿真2D机器人足球队,并在比赛中取得了较好的效果.     

股票投资中寻求马尔可夫最优决策的程序设计

本文利用（１）中的马尔可夫链，用ＦＯＲＴＲＡＮ语言设计了股票价格上扬，下跌平均时间的计算程序，利用马尔可夫决策，实现了股票买进卖出最佳策略的实用程序。     

On the General Utility of Discounted Markov Decision Processes

We will discuss an expected utility of rewards which are generated by Markov decision processes. This is applied to the optimal stopping problem with a utility treatment. Also a combined model of the decision processes and the stopping problem, called a stopped Markov decision, is considered under the utility.     

A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes

A critical issue for the application of Markov decision processes (MDPs) to realistic problems is how the complexity of planning scales with the size of the MDP. In stochastic environments with very large or infinite state spaces, traditional planning and reinforcement learning algorithms may be inapplicable, since their running time typically grows linearly with the state space size in the worst case. In this paper we present a new algorithm that, given only a generative model (a natural and common type of simulator) for an arbitrary MDP, performs on-line, near-optimal planning with a per-state running time that has no dependence on the number of states. The running time is exponential in the horizon time (which depends only on the discount factor and the desired degree of approximation to the optimal policy). Our algorithm thus provides a different complexity trade-off than classical algorithms such as value iteration—rather than scaling linearly in both horizon time and state space size, our running time trades an exponential dependence on the former in exchange for no dependence on the latter.Our algorithm is based on the idea of sparse sampling. We prove that a randomly sampled look-ahead tree that covers only a vanishing fraction of the full look-ahead tree nevertheless suffices to compute near-optimal actions from any state of an MDP. Practical implementations of the algorithm are discussed, and we draw ties to our related recent results on finding a near-best strategy from a given class of strategies in very large partially observable MDPs (Kearns, Mansour, & Ng. Neural information processing systems 13, to appear).     

连续时间Markov决策过程在呼叫接入控制中的应用

应用Markov决策过程与性能势相结合的方法,给出了呼叫接入控制的策略优化算法.所得到的最优策略是状态相关的策略,与基于节点已占用带宽决定行动的策略相比,状态相关策略具有更好的性能值,而且该算法具有很快的收敛速度.     

Optimizing Long-term Hydro-power Production Using Markov Decision Processes

Modelling the long-term operation of hydroelectric systems is one of the classic applications of Markov decision process (MDP). The computation of optimal policies, for MDP models, is usually done by dynamic programming (DP) on a discretized state space. A major difficulty arises when optimizing multi-reservoir systems, because the computational complexity of DP increases exponentially with the number of sites. This so-called 'curse of dimensionality' has so far restricted the applicability of DP to very small systems (2 or 3 sites). Practitioners have thus had to resort to other methodologies for the long-term planning, often at the expense of rigour, and without reliable error estimates. This paper surveys recent research of MDP computation, with application to hydro-power systems. Three main approaches are discussed: (i) discrete DP, (ii) numerical approximation of the expected future reward function, and (iii) analytic solution of the DP recursion.     

基于部分可观马氏决策的AUV全局路径规划

自治式水下机器人在复杂海洋环境航行时要求寻找一条从给定起始点到终止点的较优的运动路径,安全、无碰撞地绕过所有的障碍物.提出了一种基于部分可观察马尔可夫决策过程,并结合预测障碍物运动的全局路径规划新方法; 给出了部分可观马尔可夫决策的数学模型;建立了树状的分层部分可观马尔可夫决策模型,并在路径规划中应用;提出了短期预测和长期预测两种针对水下障碍物运动轨迹预测的方法;最后通过仿真实验对AUV的全局路径规划能力进行了仿真验证,为今后的实艇试验打下了很好的基础.     

基于马尔可夫决策模型的测试向量排序新方法

时延测试向量排序是降低测试功耗的有效技术。提出了基于马尔可夫决策模型的时延测试向量排序新方法。对时延测试向量进行重排序,利用基于转换频度的诱导开关方程和海明距离来定义测试向量序列的转移概率,根据转移概率决定测试向量的顺序,降低测试电路的开关翻转频率,以达到降低峰值功耗和平均功耗的目的。给出了完整的算法TVO-MDP并进行算法最优性和复杂性分析。实验结果证实了本方法的有效性。     

Meaningful Update and Repair of Markov Decision Processes for Self-Adaptive Systems

Self-adaptive systems are able to adjust their behaviour in response to environmental condition changes and are widely deployed as Internetwares.Considered as a promising way to handle the ever-growing complexity of software systems,they have seen an increasing level of interest and are covering a variety of applications,e.g.,autonomous car systems and adaptive network systems.Many approaches for the construction of self-adaptive systems have been developed,and probabilistic models,such as Markov decision processes(MDPs),are one of the favoured.However,the majority of them do not deal with the problems of the underlying MDP being obsolete under new environments or unsatisfactory to the given properties.This results in the generated policies from such MDP failing to guide the self-adaptive system to run correctly and meet goals.In this article,we propose a systematic approach to updating an obsolete MDP by exploring new states and transitions and removing obsolete ones,and repairing an unsatisfactory MDP by adjusting its structure in a more meaningful way rather than arbitrarily changing the transition probabilities to values not in line with reality.Experimental results show that the MDPs updated and repaired by our approach are more competent in guiding the self-adaptive systems’correct running compared with the original ones.