基于延迟调度策略的Reduce调度优化算法

在大规模的Hadoop集群中,良好的任务调度策略对提高数据本地性、减小网络传输开销、减少作业执行时间以及提高集群的作业吞吐量都有着重要的影响。本文针对Hadoop架构中Reduce任务的数据本地性较低问题,提出了一种基于延迟调度策略的Reduce任务调度优化算法,通过提高Reduce任务的数据本地性来减少作业执行时间以及提高作业吞吐量,该算法在Hadoop架构的Early Shuffle阶段,使用多级延迟调度策略来提高Reduce任务的数据本地性。最后重写原生公平调度器代码实现了该调度算法,并与原生公平调度器进行了对比实验分析,实验结果表明该算法明显减少了作业执行时间,提高了集群的作业吞吐量。     

公网数字集群系统的动态并发请求调度处理队列算法

针对公网数字集群系统高并发时吞吐量低、响应时间长、失败率高等问题，提出一种公网数字集群系统的动态并发请求调度处理队列（DCRSPQ）算法。首先建立请求失败率和平均响应时间的优化目标，利用请求的各类优先级系数得到每个请求的加权平均优先级；然后，采用K-means聚类算法，根据每个请求的加权平均优先级确定所属的优先级队列，将请求分类到不同优先级的队列中；最后，利用自适应资源反馈调整机制，将系统资源动态分配给各优先级队列处理器，同时动态改变各优先级队列的长度，实现各类请求的高效快速处理。仿真结果表明，与分层级赤字加权轮询队列调度（HDWRR）和基于队列长度的动态加权公平队列调度（DQLWFQ）等算法相比，DCRSPQ算法的平均响应时间能缩短23.8%以上，吞吐量可提高3.5%以上，请求成功率可提升0.4个百分点。DCRSPQ算法具有更低的平均响应时间、更高的吞吐量以及更好的请求成功率，在公网数字集群系统并发情形下能有效提升系统的处理效率。     

基于内容分发的Cache集群系统

文章设计和实现了大规模基于内容请求分发的系统。一个VS／DR或VS／TUN调度器作为系统的单一入口点，将请求分发给一组KTCPVS分发器，KTCPVS分发器通过使用中心的内容位置服务来确定处理该请求的后端服务器，然后，再将该请求分发到后端服务器，并提出基于局部性的最小负载调度算法。最后将该体系结构应用到大规模Cache集群系统中加以验证。     

基于奖惩共存收益模式的大数据作业调度器

为满足服务商获得最大收益、达到平台资源利用率最大的要求，提出一种基于奖惩共存收益模式的大数据作业调度器，该调度器中包括基于任务执行时间的确定轮数算法(TRN)和基于最大轮数的作业调度算法(MRNS)。TRN确定作业在不同奖惩阶段的Map和Reduce的最大轮数组合以及最大标准时间；MRNS选择具有局部最大收益的作业和该作业的任务最大轮数方案，制定出基于任务的作业调度策略。实验结果表明，提出的作业调度器对比已有的调度器，作业平均完成时间缩短了13.5%～25.9%、服务商收益提高了16.3%～26.4%,平台资源利用率平均提高了7.8%～10.3%,故该大数据作业调度器具有一定的高效性和可用性。     

基于累计工作量的在线大数据分析作业调度算法

针对Hadoop和Spark等大数据分析系统中无先验知识任务的高效执行问题，设计了基于累计工作量（CRW）的任务调度器CRWScheduler。该调度器根据CRW将任务在低权重队列与高权重队列间切换；在为作业分配资源时，同时考虑到作业所在的队列和其瞬时占用资源量，无需作业先验知识即显著提升系统性能。基于Apache Hadoop YARN实现了CRWScheduler原型，在28个节点的基准测试集群上的实验表明，与YARN的公平调度机制相比，作业流时间（JFT）平均降低21%，其中95百分位的作业流时间（JFT）最多降低了35%，并且在与任务级调度程序协作时可获得进一步的性能提升。     

异构云环境下的密码服务调度方法

针对云计算环境下存在密码服务请求算法种类多、资源需求差异化和节点性能异构等问题。为了提高系统的可靠性,保证服务质量,综合考虑用户请求任务和处理节点等多种因素,在作业包截止时间的基础上,通过任务映射策略完成密码服务的一级调度。设计一个基于用户优先级和任务等待时间的任务优先级调度算法实现二级调度,从而构造一种同时支持多种密码服务请求和任务动态可调整的调度系统框架,以保证云环境下任务的时效性。仿真结果表明,该系统有较好的执行效率和负载分布效果,达到设计目标。与随机法和遗传算法相比,其执行效率分别提高了17%和11%左右。     

基于云计算的网络大数据均衡调度系统设计

由于传统大数据均衡调度系统应用下的网络吞吐量较低,因此提出基于云计算的网络大数据均衡调度系统。在系统硬件方面,设计了动态调度器和网络数据管理服务器,其中动态调度器执行系统均衡调度策略来辅助系统完成数据均衡调度任务,网络数据管理服务器对系统接收的网络大数据集成管理。在系统软件方面,首先采用网络爬虫技术对网络大数据样本进行采集,在数据矢量空间中建立映射关系;其次利用模糊循环迭代控制技术对网络大数据样本进行均衡处理;最后利用云计算技术对网络大数据均衡调度策略进行计算分析,以此实现系统设计。经实验证明,应用设计系统的网络吞吐量高于传统系统。     

一种基于小数据同步写的回写I/O调度器

小数据同步写普遍存在于的各种计算机环境中,并且可以由计算机系统的不同层次软件产生,从底层操作系统一直到上层应用软件都可以生成小数据同步写请求.然而,操作系统的文件系统是以块作为最小逻辑可寻址单位,小数据写将会导致严重的写放大问题,使得系统的I/O性能大幅降低.为了解决上述问题,我们提出了一种新的I/O调度器,并将它取名为“Hitchhike”.该调度器可以识别小数据写,并通过对其他数据块中的数据进行压缩,,将小数据嵌入到压缩出来的空间中,从而将小数据和该数据块一起写入到磁盘上,以异步回写的方式完成小数据的同步写,不仅有效缓解了磁盘的写放大问题,也大大提高了小数据同步写的效率.我们基于Linux中Deadline调度器实现了Hitchhike原型系统,并利用Filebench基准测试来测试调度器在吞吐量、I/O延迟等方面的性能.通过和传统I/O调度器的性能进行比较,我们可以发现Hitchhike调度器能够显著的提高小数据同步写性能高达48.6%.     

集群系统中实现软实时服务机制的研究与评估

研究了集群架构服务器的三种实时调度策略,即集中式策略、分布式策略和主动型全分布策略。与调度功能主要集中在前端机的集中式策略比较,分布式策略将调度功能部分离散至后端节点,改善了系统的可扩展性。而主动型全分布式调度则进一步实现了调度分派功能的完全离散化,通过后端机的主动拉取事件和频率调控实现了负载平衡的自调节,极大地改善了系统的扩展性和服务效率。通过试验研究和试验表明,这三种都有效地降低了集群的响应服务请求的时间,还保证了系统的输出最大化。     

CPU+MIC异构系统中动态请求处理模型研究

为满足日益增加的服务器端的计算需求,更多的协处理器（如GPU和MIC）成为服务器端的新成员,参与服务器端计算,但是传统的服务器端软件（如Web服务器软件等）不能充分发挥协处理器的性能。为充分利用MIC的性能,提升单台Web服务器的服务质量,针对CPU+MIC的异构硬件体系提出了一种新的动态请求处理模型。该模型基于事件驱动模型和线程池模型,可将部分动态请求调度至MIC执行,并行处理动态请求,兼顾了CPU和MIC间的负载均衡。仿真实验表明,该模型在平均响应时间、吞吐量和99%响应时间等方面均优于现有的Web服务器软件模型。     

European Community policy and the market

Abstract This paper starts with some reflections on the policy considerations and priorities which are shaping European Commission (EC) research programmes. Then it attempts to position the current projects which seek to capitalise on information and communications technologies for learning in relation to these priorities and the apparent realities of the marketplace. It concludes that while there are grounds to be optimistic about the contribution EC programmes can make to the efficiency and standard of education and training, they are still too technology driven.     

一种自适应子融合集成多分类器方法

融合集成方法已经广泛应用在模式识别领域,然而一些基分类器实时性能稳定性较差,导致多分类器融合性能差,针对上述问题本文提出了一种新的基于多分类器的子融合集成分类器系统。该方法考虑在度量层融合层次之上通过对各类基多分类器进行动态选择,票数最多的类别作为融合系统中对特征向量识别的类别,构成一种新的自适应子融合集成分类器方法。实验表明,该方法比传统的分类器以及分类融合方法识别准确率明显更高,具有更好的鲁棒性。     

煤矿井下安全监控分站的设计及其仿真实现

为了设计一种具有低成本、低功耗、易操作、功能强且可靠性高的煤矿井下安全分站,针对煤矿安全生产实际,文章提出了采用MCS-51系列单片机为核心、具有CAN总线通信接口的煤矿井下安全监控分站的设计方案;首先给出煤矿井下安全监控分站的整体构架设计,然后着重阐述模拟量输入信号处理系统的设计过程,最后说明单片机最小系统及其键盘、显示、报警、通信等各个组成部分的设计;为验证设计方案的可行性与有效性,使用Proteus软件对设计内容进行仿真验证,设计的煤矿井下安全监控分站具有瓦斯、温度等模拟量参数超标报警功能和电机开停、风门开闭等开关量指示功能;仿真结果表明:设计的煤矿井下安全监控分站具有一定的实际应用价值.     

Explanation and prediction: an architecture for default and abductive reasoning

Although there are many arguments that logic is an appropriate tool for artificial intelligence, there has been a perceived problem with the monotonicity of classical logic. This paper elaborates on the idea that reasoning should be viewed as theory formation where logic tells us the consequences of our assumptions. The two activities of predicting what is expected to be true and explaining observations are considered in a simple theory formation framework. Properties of each activity are discussed, along with a number of proposals as to what should be predicted or accepted as reasonable explanations. An architecture is proposed to combine explanation and prediction into one coherent framework. Algorithms used to implement the system as well as examples from a running implementation are given.     

View-based model-driven architecture for enhancing maintainability of data access services

In modern service-oriented architectures, database access is done by a special type of services, the so-called data access services (DAS). Though, particularly in data-intensive applications, using and developing DAS are very common today, the link between the DAS and their implementation, e.g. a layer of data access objects (DAOs) encapsulating the database queries, still is not sufficiently elaborated, yet. As a result, as the number of DAS grows, finding the desired DAS for reuse and/or associated documentation can become an impossible task. In this paper we focus on bridging this gap between the DAS and their implementation by presenting a view-based, model-driven data access architecture (VMDA) managing models of the DAS, DAOs and database queries in a queryable manner. Our models support tailored views of different stakeholders and are scalable with all types of DAS implementations. In this paper we show that our view-based and model driven architecture approach can enhance software development productivity and maintainability by improving DAS documentation. Moreover, our VMDA opens a wide range of applications such as evaluating DAS usage for DAS performance optimization. Furthermore, we provide tool support and illustrate the applicability of our VMDA in a large-scale case study. Finally, we quantitatively prove that our approach performs with acceptable response times.     

A pragmatic multi-method investigation of discrepant technological events: Coping,attributions, and ‘accidental’ learning

Discrepant technological events or situations that entail a problem, a misunderstanding or a difficulty with the Information Technology (IT) being employed, are common in the workplace, and can lead to frustration and avoidance behaviors. Little is known, however, about how individuals cope with these events. This paper examines these events by using a multi-method pragmatic approach informed by coping theory. The results of two studies – a critical incident study and an experiment – serve to build and test, respectively, a theoretical model that posits that individuals use a variety of strategies when dealing with these events: they experience negative emotions, make external attributions, and adopt engagement coping strategies directed at solving the event, eventually switching to a disengagement coping strategy when they feel they have no control over the situation. Furthermore, users’ efforts may result in ‘accidental’ learning as they try to overcome the discrepant IT events through engagement coping. The paper ends with a discussion of the results in light of existing literature, future opportunities for research, and implications for practice.     

Three Process Perspectives: Organizations, Teams, and People

This paper provides the author's personal views and perspectives on software process improvement. Starting with his first work on technology assessment in IBM over 20 years ago, Watts Humphrey describes the process improvement work he has been directly involved in. This includes the development of the early process assessment methods, the original design of the CMM, and the introduction of the Personal Software Process (PSP)^SM and Team Software Process (TSP){^SM}. In addition to describing the original motivation for this work, the author also reviews many of the problems he and his associates encountered and why they solved them the way they did. He also comments on the outstanding issues and likely directions for future work. Finally, this work has built on the experiences and contributions of many people. Mr. Humphrey only describes work that he was personally involved in and he names many of the key contributors. However, so many people have been involved in this work that a full list of the important participants would be impractical.     

The Distributed Time-Triggered Simulation Scheme: Core Principles and Supporting Execution Engine

Distributed real-time simulation is a young technology field but its practice is under increasing demands. In recent years the author and his collaborators have been establishing a new approach called the distributed time-triggered simulation (DTS) scheme which is conceptually simple and easy to use but widely applicable. The concept was initiated in the course of developing a new-generation object-oriented real-time programming scheme called the time-triggered message-triggered object (TMO) programming scheme. Some fundamental issues inherent in distributed real-time simulation that were learned during recent experimental studies are discussed along with some approaches for resolving the issues. An execution engine developed to support both the TMOs engaged in control computation and the TMOs engaged in DTS is also discussed along with its possible extensions that will enable significantly larger-scale DTSs.     

Software-Konfigurationsmanagement in großen Softwareprojekten

Zusammenfassung Mit zunehmender Größe der Softwaresysteme verschärfen sich die für die Software-Herstellung typischen Probleme: Beherrschen großer Objektmengen, Erhalten der Systemkonsistenz, Kontrolle der ständigen Änderungseinflüsse und Gewährleisten einer langen Lebensdauer. Die Disziplin Konfigurationsmanagement bildet den methodischen Ansatz, diese Probleme besser zu beherrschen. Software-Konfigurationsmanagement faßt die Herstellung von Softwaresystemen als eine Abfolge von kontrollierten Änderungen an gesicherten Zwischen- und Endergebnissen auf. Dargestellt werden die Objekte und Funktionen des Software-Konfigurationsmanagements sowie die hierfür in großen Software-Projekten benötigten Methoden, Instanzen und Hilfsmittel.     

基于复小波噪声方差显著修正的SAR图像去噪

提出了一种基于复小波域统计建模与噪声方差估计显著性修正相结合的合成孔径雷达(Synthetic Aperture Radar,SAR)图像斑点噪声滤波方法。该方法首先通过对数变换将乘性噪声模型转化为加性噪声模型,然后对变换后的图像进行双树复小波变换(Dualtree Complex Wavelet Transform,DCWT),并对复数小波系数的统计分布进行建模。在此先验分布的基础上,通过运用贝叶斯估计方法从含噪系数中恢复原始系数,达到滤除噪声的目的。实验结果表明该方法在去除噪声的同时保留了图像的细节信息,取得了很好的降噪效果。