Hadoop平台下改进的推测任务调度算法

研究对比Hadoop平台下默认的推测任务调度算法和异构环境下LATE调度算法的优势和不足,提出了一种基于Hadoop集群的改进的推测任务调度算法.该算法以节点历史信息对Reduce任务各阶段比例进行动态调整和更新,并对任务实时处理速率进行局部平滑处理来提高预估任务剩余完成时间的准确性,最后采用MCP模型对备份任务有效性进行验证.通过实验结果分析可知:该算法能够有效提升备份任务成功率,减少作业完成时间.     

基于矩阵分解的协同过滤算法的并行化研究

基于矩阵分解的协同过滤算法是近几年提出的一种协同过滤推荐技术,但其每项预测评分的计算都要综合大量评分数据,同时在计算时还需要存储庞大的特征矩阵,用单一节点来进行推荐将会遇到计算时间和计算资源的瓶颈。通过对现有的基于ALS(最小二乘法)的协同过滤算法在Hadoop上并行化实现的原理和特点进行深入的研究,得到了传统的迭代式算法在Hadoop上运算效率不高的原因。根据迭代式MapReduce思想,提出了循环感知任务调度算法、缓存静态数据、任务循环控制、迭代终止条件检测等方法。通过在Netflix数据集上的实验表明,迭代式MapReduce思想提高了基于ALS的协同过滤算法的并行化计算的效率。     

基于Flink的任务调度策略

针对大数据流式计算平台拓扑中因各关键节点上任务间不同类型的通信方式导致的通信开销较大问题,提出一种Flink环境下的任务调度策略。通过各任务间数据流大小确定拓扑边权重,将有向无环图转化为拓扑关键路径模型,在保证关键路径上节点负载差异较小的同时,最小化关键任务的节点间通信开销。实验结果表明,该算法与Flink平台现有的任务调度策略相比,在WordCount和TwitterSentiment作业执行过程中计算平均时延降低了13.09%,有效提升了系统性能。     

异构环境下降低慢任务抖动的调度算法——DPST

为解决在异构计算环境中现有的云计算负载均衡算法存在的慢任务频繁抖动的问题,提出了一种能减低慢任务调度抖动概率的算法--DPST算法。首先通过定义一种异构计算节点中异构任务的能力度量,对执行异构任务的节点处理能力进行了归一化;然后通过引入节点能力预判机制,降低慢任务无效调度的次数;并且利用慢任务和慢节点双队列机制,提高了调度效率。实验结果表明,DPST相对于Hadoop平台在异构环境下任务调度的抖动次数下降了40%以上。由于有效降低了任务调度的抖动次数,在异构环境中DPST算法能明显地缩短任务的平均响应时间并提高系统的吞吐量。     

基于任务进度感知的异构Hadoop云平台任务调度方案

针对异构Hadoop云计算平台的任务调度问题,对Hadoop 推测执行调度和LATE调度方案进行研究,提出一种基于任务进度感知的自适应任务调度方案。首先,根据当前计算节点上的任务进度情况,估计任务近似完成时间(ATE),以此确定掉队者(Straggler)任务。然后,以平均节点任务进度的25%为阈值,将节点分为慢节点和快节点。当Straggler后备任务达到一定阈值时,将其优先分配到快节点中执行。实验结果表明,提出的方案能够为异构Hadoop平台合理分配任务,有效降低了任务完成时间和响应延迟,同时提高了平台吞吐量。     

基于延迟调度策略的Reduce调度优化算法

在大规模的Hadoop集群中,良好的任务调度策略对提高数据本地性、减小网络传输开销、减少作业执行时间以及提高集群的作业吞吐量都有着重要的影响。本文针对Hadoop架构中Reduce任务的数据本地性较低问题,提出了一种基于延迟调度策略的Reduce任务调度优化算法,通过提高Reduce任务的数据本地性来减少作业执行时间以及提高作业吞吐量,该算法在Hadoop架构的Early Shuffle阶段,使用多级延迟调度策略来提高Reduce任务的数据本地性。最后重写原生公平调度器代码实现了该调度算法,并与原生公平调度器进行了对比实验分析,实验结果表明该算法明显减少了作业执行时间,提高了集群的作业吞吐量。     

Map/Reduce下快速剪枝算法在复杂任务调度中的应用

云环境下传统任务分配与调度算法对于复杂任务调度的整体效率较低,为了提高Map/Reduce对复杂任务分配调度的整体效率,提出了一种基于任务处理时间的快速剪枝算法。该算法首先将复杂任务按照任务依赖关系大小进行最佳拓扑排序,使任务按顺序执行,从而提高调度准确率。然后使用节点处理任务的预测时间与节点处理能力的比值作为子任务在每个节点的处理时间进行量化建模,建立任务和处理时间的度量矩阵,通过采用按阶剪枝方法逐渐缩小任务分配规模,对N个节点处理N个任务的分配问题,进行N-1次操作可获得任务分配的最优解。运用Hadoop平台进行实验验证,从任务调度效率与资源使用率角度将剪枝算法与公平调度算法、遗传算法和GRAPHENE算法进行对比验证。实验结果表明剪枝算法能明显提高任务调度的整体效率,充分利用各节点的计算能力提高Map/Reduce调度效率。     

基于移动雾计算的线上订单任务资源优化分配

在物联网环境下,线上订单处理的可移动设备存在计算能力有限、内存限制和计算延迟等缺点。通过引入移动雾计算（Mobile Fog Computing）优化线上订单任务,运用模糊聚类算法对雾节点进行聚类,运用贪心算法对聚类后的任务资源进行优化调度,使用Visual Studio工具对贪心算法进行调节和运行,得出任务调度匹配结果,提高平台运营效率。结合丹尼斯某超市线上订单的实际数据,对雾节点任务资源调度执行时间进行分析,结果表明该算法具有可行性和实用性。     

一种异构环境下的基于MapReduce任务调度改进机制

针对在异构环境下采用现有MapReduce任务调度机制可能出现各计算节点间数据迁移和系统资源分配难以管理的问题, 提出一种动态的任务调度机制来改善这些问题。该机制先根据节点的计算能力按比例放置数据, 然后通过资源预测方法估计异构环境下MapReduce任务的完成时间, 并根据完成时间计算任务所需的资源。实验结果表明, 该机制提高了异构环境下任务的数据本地性比例, 且能动态地调整资源分配, 以保证任务在规定时间内完成, 是一种有效可行的任务调度机制。     

异构环境下基于双重预取的Hadoop调度算法

Hadoop处理海量数据时,无论是Map任务还是Reduce任务都需要耗费大量的时间传输数据,故提出一种基于双重预取的调度算法;该算法通过估算节点上任务执行的进度来预测Map任务的执行节点,然后通知节点提前预取所需的数据,并且在Map任务完成的数量达到预定值时,开始为Reduce任务预取部分数据;由于在异构的环境下集群中节点的性能各不相同,为此采取了改进的预测模型,以提高任务进度判断的准确性;实验证明,本算法在作业响应时间等方面优于现有的调度算法。     

Only connect: teaching, technology and telesis

Abstract This paper describes an approach to the design of interactive multimedia materials being developed in a European Community project. The developmental process is seen as a dialogue between technologists and teachers. This dialogue is often problematic because of the differences in training, experience and culture between them. Conditions needed for fruitful dialogue are described and the generic model for learning design used in the project is explained.     

European Community policy and the market

Abstract This paper starts with some reflections on the policy considerations and priorities which are shaping European Commission (EC) research programmes. Then it attempts to position the current projects which seek to capitalise on information and communications technologies for learning in relation to these priorities and the apparent realities of the marketplace. It concludes that while there are grounds to be optimistic about the contribution EC programmes can make to the efficiency and standard of education and training, they are still too technology driven.     

一种自适应子融合集成多分类器方法

融合集成方法已经广泛应用在模式识别领域,然而一些基分类器实时性能稳定性较差,导致多分类器融合性能差,针对上述问题本文提出了一种新的基于多分类器的子融合集成分类器系统。该方法考虑在度量层融合层次之上通过对各类基多分类器进行动态选择,票数最多的类别作为融合系统中对特征向量识别的类别,构成一种新的自适应子融合集成分类器方法。实验表明,该方法比传统的分类器以及分类融合方法识别准确率明显更高,具有更好的鲁棒性。     

Avoiding semantic and temporal gaps in developing software intensive systems

Development of software intensive systems (systems) in practice involves a series of self-contained phases for the lifecycle of a system. Semantic and temporal gaps, which occur among phases and among developer disciplines within and across phases, hinder the ongoing development of a system because of the interdependencies among phases and among disciplines. Such gaps are magnified among systems that are developed at different times by different development teams, which may limit reuse of artifacts of systems development and interoperability among the systems. This article discusses such gaps and a systems development process for avoiding them.     

Designing economic np control charts: A programmed simulation approach

This paper presents control charts models and the necessary simulation software for the location of economic values of the control parameters. The simulation program is written in FORTRAN, requires only 10K of main storage, and can run on most mini and micro computers. Two models are presented - one describes the process when it is operating at full capacity and the other when the process is operating under capacity. The models allow the product quality to deteriorate to a further level before an existing out-of-control state is detected, and they can also be used in situations where no prior knowledge exists of the out-of-control causes and the resulting proportion defectives.     

The development of robot art

Going through a few examples of robot artists who are recognized worldwide, we try to analyze the deepest meaning of what is called “robot art” and the related art field definition. We also try to highlight its well-marked borders, such as kinetic sculptures, kinetic art, cyber art, and cyberpunk. A brief excursion into the importance of the context, the message, and its semiotics is also provided, case by case, together with a few hints on the history of this discipline in the light of an artistic perspective. Therefore, the aim of this article is to try to summarize the main characteristics that might classify robot art as a unique and innovative discipline, and to track down some of the principles by which a robotic artifact can or cannot be considered an art piece in terms of social, cultural, and strictly artistic interest. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

Explanation and prediction: an architecture for default and abductive reasoning

Although there are many arguments that logic is an appropriate tool for artificial intelligence, there has been a perceived problem with the monotonicity of classical logic. This paper elaborates on the idea that reasoning should be viewed as theory formation where logic tells us the consequences of our assumptions. The two activities of predicting what is expected to be true and explaining observations are considered in a simple theory formation framework. Properties of each activity are discussed, along with a number of proposals as to what should be predicted or accepted as reasonable explanations. An architecture is proposed to combine explanation and prediction into one coherent framework. Algorithms used to implement the system as well as examples from a running implementation are given.     

Three Process Perspectives: Organizations, Teams, and People

This paper provides the author's personal views and perspectives on software process improvement. Starting with his first work on technology assessment in IBM over 20 years ago, Watts Humphrey describes the process improvement work he has been directly involved in. This includes the development of the early process assessment methods, the original design of the CMM, and the introduction of the Personal Software Process (PSP)^SM and Team Software Process (TSP){^SM}. In addition to describing the original motivation for this work, the author also reviews many of the problems he and his associates encountered and why they solved them the way they did. He also comments on the outstanding issues and likely directions for future work. Finally, this work has built on the experiences and contributions of many people. Mr. Humphrey only describes work that he was personally involved in and he names many of the key contributors. However, so many people have been involved in this work that a full list of the important participants would be impractical.     

基于复小波噪声方差显著修正的SAR图像去噪

提出了一种基于复小波域统计建模与噪声方差估计显著性修正相结合的合成孔径雷达(Synthetic Aperture Radar,SAR)图像斑点噪声滤波方法。该方法首先通过对数变换将乘性噪声模型转化为加性噪声模型,然后对变换后的图像进行双树复小波变换(Dualtree Complex Wavelet Transform,DCWT),并对复数小波系数的统计分布进行建模。在此先验分布的基础上,通过运用贝叶斯估计方法从含噪系数中恢复原始系数,达到滤除噪声的目的。实验结果表明该方法在去除噪声的同时保留了图像的细节信息,取得了很好的降噪效果。     

How do children do mathematics with LOGO?

Abstract  This paper considers some results of a study designed to investigate the kinds of mathematical activity undertaken by children (aged between 8 and 11) as they learned to program in LOGO. A model of learning modes is proposed, which attempts to describe the ways in which children used and acquired understanding of the programming/mathematical concepts involved. The remainder of the paper is concerned with discussing the validity and limitations of the model, and its implications for further research and curriculum development.