基于蜜罐技术的负载均衡算法研究

现有负载均衡算法难以适应蜜场环境下网络流量具有突发性强、流量大等特点.分析了当前流行的几种基本负载均衡算法,提出了一种面向蜜场环境的负载均衡算法,通过设计预警收敛流量分配策略、主动抢占式反馈策略以及基于排队模型的等待队列动态调整策略,有效地实现了蜜场环境下的负载均衡.     

基于对象存储的负载均衡存储策略

对象存储文件系统中将大数据文件分片存储到多个存储节点上,以获取更好的并行I/O性能,提高系统吞吐率.现有对象存储文件系统的存储策略并未充分考虑存储对象本身负载的动态变化,不利于提高系统资源利用率.针对此问题,考虑存储对象的空间及I/O等负载实时变化,提出了一种简单、灵活、高效的负载均衡存储策略,并对该策略进行了实现.实验结果表明,该策略能有效提高对象存储系统资源的利用率和吞吐率,保证对象存储文件系统高效的读写性能.     

面向SLA的云计算负载均衡策略

云计算负载均衡是保障SLA协议的关键问题之一;针对云计算负载均衡问题,提出一种面向SLA的负载均衡策略;该策略引入人工神经网络思想,建立负载均衡模型,采用单层感知器算法(SLPA)将虚拟机负载状态进行分类,然后利用结合了动态加权轮询算法的BP神经网络算法(BPNNA-DWRRA)有针对性地对虚拟机负载权重进行预测更新,最后将任务调度到最小权重所对应的可行虚拟机上;应用CloudSim进行仿真实验,结果表明了该策略的可行性,同时,相比加权最小链接算法和粒子群算法,该策略的平均响应时间分别节省了43.6%和22.5%,SLA违反率分别降低了20.7%和14.4%;因此,所提策略在响应用户任务时,请求响应时间短,SLA违反率低,保障了SLA。     

面向并行网络流量分类的负载均衡算法

为了解决高速网络流量分类系统的性能瓶颈问题,提出了一种并行网络流量分类系统负载均衡算法。该算法由静态预分配和动态自适应调整两部分组成,采用基于Hash流表实现负载的静态预分配,根据处理节点的动态反馈对Hash流表进行重映射。通过实验和静态Hash算法以及SHI算法进行了对比,实验结果表明,该算法负载均衡度好、丢包率小、流重映射率低,能够满足并行网络流量分类系统负载均衡的要求。     

SNFS并行文件系统负载均衡技术的研究与实现

在大规模集群系统的并行运算环境中,I/O效率一直是影响系统整体性能的关键因素,并行文件系统技术是目前解决I/O性能瓶颈的有效途径之一。介绍当前并行文件系统的发展现状以及并行文件系统的类型,阐述SNFS并行文件系统的架构以及负载均衡DLC(分布式LAN客户端)技术的实现原理,并给出大规模集群系统环境中SNFS文件系统负载均衡技术的实现方法,最后,通过实际应用分析说明该技术在提升I/O性能上的优势。     

基于MapReduce的高维数据频繁项集挖掘

传统的数据挖掘算法在面向大规模高维数据的挖掘过程中,存在数据特征捕捉准确率低、节点负载不均衡、数据交互频繁、频繁项集紧凑化程度低等问题。提出基于MapReduce的并行挖掘算法PARDG-MR,结合高维数据特征,设计基于维度粒化算法和负载均衡算法的DGPL策略,并对数据进行预处理,以解决高维复杂数据特征属性捕捉困难及数据划分中节点负载不均衡的问题。通过构建基于PJPFP-Tree树的频繁项集并行挖掘策略PARM,实现频繁项集的并行化分组过程,从而提高数据处理的运行效率。在此基础上,提出基于剪枝前缀推论的整合节点剪枝算法PJPFP,提高频繁项集挖掘过程中的剪枝效率,增强频繁项集的紧凑化程度。在Webdocs、NDC、Gisette 3个数据集上的实验结果表明,相比PFP-growth、PWARM、MRPrePost算法,该算法的运行时间平均缩短了约20%,能够有效提高数据挖掘效率且降低内存空间。     

基于Winograd卷积的并行深度卷积神经网络优化算法

针对并行深度卷积神经网络算法在大数据环境下存在冗余特征计算过多、卷积运算性能不足和参数并行化合并效率低等问题，提出了基于Winograd卷积的并行深度卷积神经网络优化算法。首先，该算法提出基于余弦相似度与归一化互信息的特征过滤策略，通过先筛选后融合的方式消除了通道间对于冗余特征的计算，以此解决了冗余特征计算过多的问题；然后，结合MapReduce提出了并行Winograd卷积策略，通过使用并行化Winograd卷积运算替换传统卷积运算的方式来提升卷积运算的性能，以此解决了卷积运算性能不足的问题；最后，提出基于任务迁移的负载均衡策略，通过动态负载迁移的方式来均衡集群中各节点之间的负载，降低了集群总体的平均反应时长，以此解决了参数并行化合并效率低的问题。实验表明，WP-DCNN算法显著降低了DCNN在大数据环境下的训练代价，而且对并行DCNN的训练效率也有大幅提升。     

MapReduce框架下一种负载均衡的Top-k连接查询算法

针对传统Top-k连接查询算法在处理海量数据时的时效问题,提出一种基于MapReduce框架的负载均衡的并行Top-k连接查询算法(P-TKJ)。使用直方图形式来存储数据,有助于提高CPU的利用率。同时融入了提前终止策略和磁盘数据的选择性访问,以便提高对HDFS数据访问的性能。另外,提出了一种基于最长处理时间优先(LPT)算法的负载均衡策略来均衡Reduce任务,以此设计出高效的并行Top-k连接算法。一个集群实验结果表明,该方法能够有效缩短算法的执行时间。     

国防科大计算机学院工学博士学位论文摘要

面向并行科学计算应用的并行文件系统作者:卢凯(1999.10)导师:金士尧教授　　本文针对并行科学计算应用,集中研究了在大规模并行处理系统上实现高性能并行文件系统的若干关键技术。本文深入研究了当前国际上的各种并行文件系统技术,分析归纳了其优缺点。在此基础上,我们从并行科学计算应用的I/O访问模式、面向并行科学计算应用的并行I/O服务模型及相关界面、并行I/O服务算法和cache预取算法等方面提出了提高其服务性能的若干技术。本文通过动、静态相结合的分析技术分析了若干典型并行科学计算应用的I/O访问特性,提取了并行科学计算应用…     

一种异构NOWs上负载均衡的适应式并行调度算法

分析了并行作业调度算法，针对异构工作站群机系统(Network of Workstations，NOWs)提出了一种负载均衡的适应式划分(Adaptive Partition，AP)策略的并行调度算法，它利用了短作业需求优先(Shortest Demand First，SDF)的策略，主要思想是让系统中所有作业尽可能均分同时结束，以达到均衡负载。实验结果证明该算法公平和高效。     

集群计算并行I/O中动态文件分配策略

针对集群计算系统中并行I／O文件分配问题进行了研究,提出一种新的动态文件分配算法一启发式负载与服务时间变化综合平衡策略。该策略将所需要的访问服务时间相似的文件分配到集群系统的同一结点磁盘上,并保证每个磁盘上的负载值不超过负载平衡的极限值,从而在所有结点磁盘间既基本实现负载均衡,又使每个磁盘上文件间的服务时间变化最小化。实验结果表明了该策略在性能提高方面的有效性。     

基于动态副本技术的云存储负载均衡研究

以系统总响应时间最小化为目标,以文件热度为依据,提出了一种多时间窗负载均衡策略。在计算文件热度时,不仅考虑了访问的次数和大小,还将I/O访问时序引入到文件热度统计中,该方法能有效控制短时间突发性数据访问导致的不必要副本创建。在多时间窗负载均衡策略中,设置了三种不同大小的时间窗口,分别实现了存储节点负载均衡、文件副本的负载均衡以及低热度文件多余副本的删除工作。实验数据表明,多时间窗负载均衡策略能显著降低I/O访问响应时间。     

On the Design of Distributed Object Placement and Load Balancing Strategies in Large-Scale Networked Multimedia Storage Systems

In a large-scale multimedia storage system (LMSS) where client requests for different multimedia objects may have different demands, the placement and replication of the objects is an important factor, as it may result in an imbalance in server loading across the system. Since replication management and load balancing are all the more crucial issues in multimedia systems, in the literature, these problems are handled by centralized servers. Each object storage server (OSS) responds to the requests that come from the centralized servers independently and has no communication with other OSSs in the system. In this paper, we design a novel distributed load balancing strategy for LMSS, in which OSSs can cooperate to achieve higher performance. Such OSS modeled as an M/G/m system can replicate the objects to and balance the requests among other servers to achieve a near-optimal average waiting time (AWT) of the requests in the system. We validate the performance of the system via rigorous simulations with respect to several influencing factors and prove that our proposed strategy is scalable, flexible, and efficient for real-life applications.     

SPIFFI-a scalable parallel file system for the Intel Paragon

This paper presents the design and performance of SPIFFI, a scalable high-performance parallel file system intended for use by extremely I/O intensive applications including “Grand Challenge” scientific applications and multimedia systems. This paper contains experimental results from a SPIFFI prototype on a 64 node/64 disk Intel Paragon. The results show that SPIFFI provides high performance and linear scaleup on real hardware. The paper also explains how shared file pointers (i.e., file pointers that are shared by multiple processes) can simplify the design of a parallel application. By sequentializing I/O accesses and by providing dynamic I/O load balancing, a shared file pointer may even improve an application's performance. This paper also presents the predictions of a SPIFFI simulator that we validated using the prototype. The simulator results show that SPIFFI continues to provide high performance even when it is scaled to configurations with as many as 128 disks or 256 compute nodes     

分布式文件系统元数据服务的负载均衡框架

请求负载均衡,是分布式文件系统元数据管理需要面对的核心问题.以最大化元数据服务器集群吞吐量为目标,在已有元数据管理层之上设计实现了一种分布式缓存框架,专门管理热点元数据,均衡不断变化的负载.与已有的元数据负载均衡架构相比,这种两层的负载均衡架构灵活度更高,对负载的感知能力更强,并且避免了热点元数据重新分布、迁移引起的元数据命名空间结构被破坏的情况.经观察分析,元数据尺寸小、数量大,预取错误元数据带来的代价远远小于预取错误数据带来的代价.针对元数据的以上鲜明特点,提出一种元数据预取策略和基于预取机制的元数据缓存替换算法,加强了上述分布式缓存层的性能,这种两层的元数据负载均衡框架同时考虑了缓存一致性的问题.最后,在一个真实的分布式文件系统中验证了框架及方法的有效性.     

A novel distributed architecture of large-scale multimedia storage system using autonomous object-based storage devices

In a large-scale multimedia storage system (LMSS) where the user requests for different multimedia objects may have different demands, placement and replication of the objects is an important factor, as it may result in an imbalance in loading across the system. Since replica management and load balancing is a crucial issue in multimedia systems, normally this problem is handled by centralized servers, e.g., metadata servers (MDS) in distributed file systems. Each object-based storage device (OSD) responds to the requests coming from the centralized servers independently and has no communication with other OSDs among the system. In this paper, we design a novel distributed architecture of LMSS, in which the OSDs have some kind of intelligences and can cooperate to achieve a high performance. Such an OSD, named as autonomous object-based storage device (AOSD), can replicate the objects to and balance the requests among other AOSDs, and handle fail-over and recovery autonomously. In the proposed architecture, we move the request balancing from centralized MDS to AOSDs and make the system more scalable, flexible, and robust. Based on the proposed architecture, we propose two different object replication and load balancing algorithms, named as “Minimum Average Waiting Time” (MAWT) and “One of the Best Two Choices” (OBTC), respectively. We validate the performance of the algorithms via rigorous simulations with respect to several influencing factors. Our findings conclusively demonstrate that the proposed architecture minimizes the average waiting time and at the same time carries out load balancing across servers.     

并行文件系统扩容与I/O节点负载平衡策略

在规模并行处理系统ＭＰＰ商业化应用中,并行文件系统容量与Ｉ／Ｏ节点的负载平衡一直是挖掘整个系统并行性,进而提高系统性能的关键问题之一。本文从分析Ｐａｒａｇｏｎ大规模并行处理系统的体系结构和并行文件系统的体系结构出发,给出了并行文件系统的扩容策略,使用ＭＰＰ系统的Ｉ／Ｏ节点平衡负载理论模型,给出了实际工程应用中的负载平衡策略。     

一种MongoDB应用优化策略

为了解决不断增长的文件存储需求,和高流量高并发的访问量,增强系统的响应性能,提出一种Web应用优化策略,通过MongoDB GridFS对网站文件存储容量、可用性和可靠性进行扩展,并且通过Nginx和Keepalived,对后台访问进行负载均衡和双机热备,优化并发性能.实验表明,当并发访问数上升至80以上时,平均访问响应时间缩短9%.文件通过Nginx Gridfs进行高并发上传时非常稳定,在较大文件下载时比直接通过本地文件系统EXT4下载速度更高.     

Multimedia Caching Strategies for Heterogeneous Application and Server Environments

In a multimedia system, storage and bandwidth are critical resources since any presentation requires a large volume of data to be delivered in real-time. Caching of multimedia documents in local storage can alleviate large retrieval bandwidth requirements. An important requirement for a multimedia caching policy is to guarantee continuous delivery even when a stream is served from cache. It should also cope with dynamic changes in workload and heterogeneity arising from large and small multimedia files. The proposed Generalized Interval Caching (GIC) policy, that caches intervals between successive streams of a large file as well as entire small files, satisfies all the above criteria. A caching policy needs to cope with additional challenges in a large scale distributed multimedia environment consisting of many heterogeneous servers. The issues include a) routing of requests to ensure good cache hits in each server, and b) balancing of loads across servers. For routing of requests, we introduce the notion of an asset group and propose an affinity routing policy based on this concept. Finally, we adapt the GIC policy for load balancing across servers.     

Dynamic load-balancing of image processing applications on clusters of workstations

Cluster-based computing, which exploits the aggregate power of a network of workstations, has drawn increasing attention from the parallel processing community. The main problem with this computing environment is the permanently changing workload of individual workstations which makes the efficiency and the execution time of parallel applications unpredictable. In this paper, we introduce an efficient load balancing scheme which aims at dynamically balancing the workload of data parallel applications in this computing environment. Simulation and experimental studies of our load balancing strategy are performed under various load situations and it is shown that it can effectively balance the workload among the workstations involved. Further, it was shown that a significant improvement in computing performance can be achieved when using our load balancing strategy as compared to the case where no load balancing is applied, particularly under a heavily loaded system.