Batching and dynamic allocation techniques for increasing the stream capacity of an on-demand media server

A server for an interactive distributed multimedia system may require thousands of gigabytes of storage space and high I/O bandwidth. In order to maximize system utilization, and thus minimize cost, the load must be balanced among the server's disks, interconnection network and scheduler. Many algorithms for maximizing retrieval capacity from the storage system have been proposed. This paper presents techniques for improving server capacity by assigning media requests to the nodes of a server so as to balance the load on the interconnection network and the scheduling nodes. Five policies for dynamic request assignment are developed. An important factor that affects data retrieval in a high-performance continuous media server is the degree of parallelism of data retrieval. The performance of the dynamic policies on an implementation of a server model developed earlier is presented for two values of the degree of parallelism.     

Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems

Big data is an emerging term in the storage industry, and it is data analytics on big storage, i.e., Cloud-scale storage. In Cloud-scale (or EB-scale) file systems, load balancing in request workloads across a metadata server cluster is critical for avoiding performance bottlenecks and improving quality of services.Many good approaches have been proposed for load balancing in distributed file systems. Some of them pay attention to global namespace balancing, making metadata distribution across metadata servers as uniform as possible. However, they do not work well in skew request distributions, which impair load balancing but simultaneously increase the effectiveness of caching and replication. In this paper, we propose Cloud Cache (C²), an adaptive and scalable load balancing scheme for metadata server cluster in EB-scale file systems. It combines adaptive cache diffusion and replication scheme to cope with the request load balancing problem, and it can be integrated into existing distributed metadata management approaches to efficiently improve their load balancing performance. C² runs as follows: 1) to run adaptive cache diffusion first, if a node is overloaded, loadshedding will be used; otherwise, load-stealing will be used; and 2) to run adaptive replication scheme second, if there is a very popular metadata item (or at least two items) causing a node be overloaded, adaptive replication scheme will be used, in which the very popular item is not split into several nodes using adaptive cache diffusion because of its knapsack property. By conducting performance evaluation in trace-driven simulations, experimental results demonstrate the efficiency and scalability of C².     

A novel distributed architecture of large-scale multimedia storage system using autonomous object-based storage devices

In a large-scale multimedia storage system (LMSS) where the user requests for different multimedia objects may have different demands, placement and replication of the objects is an important factor, as it may result in an imbalance in loading across the system. Since replica management and load balancing is a crucial issue in multimedia systems, normally this problem is handled by centralized servers, e.g., metadata servers (MDS) in distributed file systems. Each object-based storage device (OSD) responds to the requests coming from the centralized servers independently and has no communication with other OSDs among the system. In this paper, we design a novel distributed architecture of LMSS, in which the OSDs have some kind of intelligences and can cooperate to achieve a high performance. Such an OSD, named as autonomous object-based storage device (AOSD), can replicate the objects to and balance the requests among other AOSDs, and handle fail-over and recovery autonomously. In the proposed architecture, we move the request balancing from centralized MDS to AOSDs and make the system more scalable, flexible, and robust. Based on the proposed architecture, we propose two different object replication and load balancing algorithms, named as “Minimum Average Waiting Time” (MAWT) and “One of the Best Two Choices” (OBTC), respectively. We validate the performance of the algorithms via rigorous simulations with respect to several influencing factors. Our findings conclusively demonstrate that the proposed architecture minimizes the average waiting time and at the same time carries out load balancing across servers.     

基于JCF中间件的负载预测与过载迁移的融合算法

针对多种负载预测方法的适用场景进行了研究,提出了一种负载预测与过载迁移的融合算法。该算法提前对周期内无法提取到的负载情况进行预测且对超过负载阈值的服务器进行告警,关闭该进程将服务请求分发给轻负载服务器。通过提前预测与负载迁移的有机结合,解决了一个周期内无法获取各个节点的实际负载的问题,负载预测不准确的问题,以及可能出现的部分服务器过载、甚至崩溃的问题,最终达到了系统的负载均衡。在JCF（Java Component Framework）中间件平台上的实验结果表明本文算法优于静态加权轮询算法和动态加权轮询算法,在提高均衡效率、增加集群系统的吞吐量、降低服务请求响应时间多方面有着显著效果,在实际应用中有较大的价值。     

基于动态区间映射的数据对象布局算法

高效、可伸缩的数据管理在大规模分布存储系统中日益重要,关键是需要一种能够自动适应存储节点增加或减少的灵活、均衡和可伸缩的数据对象布局与定位方法.提出了一种基于动态区间映射的数据对象布局算法,在均衡数据分配和最少迁移数据方面都是统计意义上最优的,并且支持按照存储节点的权重分配数据和任意的数据对象副本.     

On the Design of Distributed Object Placement and Load Balancing Strategies in Large-Scale Networked Multimedia Storage Systems

In a large-scale multimedia storage system (LMSS) where client requests for different multimedia objects may have different demands, the placement and replication of the objects is an important factor, as it may result in an imbalance in server loading across the system. Since replication management and load balancing are all the more crucial issues in multimedia systems, in the literature, these problems are handled by centralized servers. Each object storage server (OSS) responds to the requests that come from the centralized servers independently and has no communication with other OSSs in the system. In this paper, we design a novel distributed load balancing strategy for LMSS, in which OSSs can cooperate to achieve higher performance. Such OSS modeled as an M/G/m system can replicate the objects to and balance the requests among other servers to achieve a near-optimal average waiting time (AWT) of the requests in the system. We validate the performance of the system via rigorous simulations with respect to several influencing factors and prove that our proposed strategy is scalable, flexible, and efficient for real-life applications.     

缓存与迁移策略下对等网络节点负载控制算法

在非结构化对等(P2P)网络中,节点“搭便车”行为以及用户查询不均匀会造成网络节点负载分布严重失衡。为此,提出一种节点负载控制算法,采用缓存和链路迁移策略,将重载节点中的剩余负载向其他轻载节点转移,同时在链路节点中缓存网络中的热门文件资源。实验结果表明,在用户查询服从Zipf分布的环境下,该算法能使网络中节点负载达到较好的均衡,降低系统整体负载。     

OCFS：一种基于对象存储结构的可伸缩高性能集群文件系统

提出了一种基于确定性随机分布算法分布元数据和数据对象的可伸缩集群文件系统结构。其中目录路径属性与目录对象分离的元数据管理方法，在提高系统性能、均衡元数据分布和减少元数据迁移等方面具有明显优势。提出的基于动态区间映射的数据对象布局算法，支持权重分布和副本，在均衡数据分布和最少迁移数据方面都具有统计意义上的最优性，有效解决了动态存储系统的数据均衡分布与可伸缩性问题。     

Lossless data transmission for Internet of things application over wireless multimedia sensor networks using enhanced and optimal path scheduling approach to maximizing the quality of service

The congestion of packet forwarding between a source and destination is challenging on downlink transmission in the entire file (ex. Audio and Video). Whenever file is been uploaded to the server, a user requests for file where server transmits it without knowledge of user's bandwidth, which is a major, cause of packet loss or time duration in the receiver end. To accumulate the better solution, Enhanced and Optimal Path Scheduling Approach (EOPSA) designs to find optimal path scheduling for multimedia data transmission in multimedia sensor network over cloud server using IoT devices. EOPSA studied the multisource video-on-demand streaming in multimedia sensor networks. The method introduced a heuristic distributed protocol to find optimal route for multimedia data transmissions. Efficient way to identify the bandwidth before the transmission ensures link establishment between sender and receiver. Here, the capture of bandwidth helps to check user's system capability to forward requested media data. Based on experiment evaluation, EOPSA improves 0.20 packet delivery ratio, 130 throughput, 0.20 second average delay and 14 communication overhead for 15, 25, 50, 75, and 100 nodes compared than conventional methods.     

Effects of User Request Patterns on a Multimedia Delivery System

Because of their size, service times, and drain on server resources, multimedia objects require specialized replication systems in order to meet demand and ensure content availability. We present a novel method for creating replication systems where the replicated objects' sizes and/or per-object service times are large. Such replication systems are well-suited to delivering multimedia objects on the Internet. Assuming that user request patterns to the system are known, we show how to create replication systems that distribute read load to servers in proportion to their contribution to system capacity and experimentally show the positive load distribution properties of such systems. However, when user request patterns differ from what the system was designed for, system performance will be affected. Therefore, we also report on results that reveal (i) how server loads are affected and (ii) the impact two system design parameters (indicators of a system's load distribution qualities) have on server load when request patterns differ from that for which a system was designed.     

一种基于文件支持度的动态副本管理机制

在大规模分布式存储系统的容错技术中,数据副本管理是一种重要机制.针对网络环境中的动态副本管理需求,建立一种文件支持度指标及其动态计算模型.该模型通过周期性数据采集,利用文件支持度的自相关性,结合文件上一采集周期访问量、访问量占比、被访问数据量以及文件级别等参数,构建了能够较准确描述文件的动态副本需求状态模型.通过动态适应性的参数调整以适应变化的负载状态,使副本管理决策尽可能反映系统实际状态.在此基础上设计了数据结点负载均衡、副本调整、副本清理等相关算法,实现了动态副本管理的目标.通过实验验证了所设计的动态副本管理机制的有效性.     

Faster Web page allocation with neural networks

To maintain quality of service, some heavily trafficked Web sites use multiple servers, which share information through a shared file system or data space. The Andrews file system (AFS) and distributed file system (DFS), for example, can facilitate this sharing. In other sites, each server might have its own independent file system. Although scheduling algorithms for traditional distributed systems do not address the special needs of Web server clusters well, a significant evolution in the computational approach to artificial intelligence and cognitive engineering shows promise for Web request scheduling. Not only is this transformation - from discrete symbolic reasoning to massively parallel and connectionist neural modeling - of compelling scientific interest, but also of considerable practical value. Our novel application of connectionist neural modeling to map Web page requests to Web server caches maximizes hit ratio while load balancing among caches. In particular, we have developed a new learning algorithm for fast Web page allocation on a server using the self-organizing properties of the neural network (NN).     

多媒体文件服务器的磁盘带宽分配策略

讨论多媒体文件服务器磁盘I/O带宽的分配策略问题,提出一种根据不同应用请求的负载变化情况对服务器磁盘带宽进行自适应分配的算法。该算法由负载检测模块、带宽管理模块组成,提高了多媒体文件服务器的易管理性,在短暂的过载情况下保持了服务器的稳定性,为更多的软实时请求提供服务。     

VOD服务器集群中的改进SLF存储调度策略

在VOD服务器集群中,存储调度策略是影响整个系统存储容量和总并发数的关键技术之一.针对现有存储调度策略中最小负载优先（SLF）副本放置算法调整代价过高的问题,提出了一种改进SLF算法.算法以最小化负载不平衡度和最小化副本调整代价为目标,在放置过程中充分利用当前已经存储的副本,降低副本调整的代价.仿真实验表明,基于改进SLF算法的存储调度策略可以最小化负载不平衡度,降低了存储调度的调整代价,同时提高了系统的用户请求接受概率.     

Efficient, balanced data placement algorithm in scalable storage clusters

Data distribution and load balancing become increasingly important in large-scale distributed storage system. This paper -focuses on the problem of designing an optimal, self-adaptive strategies for balanced distribution and reorganization of replicated objects among a dynamically heterogeneous nodes, and presents a novel decentralized algorithm, Dynamic Interval Mapping, which maps replicated objects to a scalable collection of nodes, it distributes objects to nodes optimally, redistributing minimum amount of objects when new nodes are added or existing nodes are removed to maintain the balanced distribution. It supports weighted allocation and guarantees that replicas of a particular object are not placed on the same node. The time complexity and storage requirements are superior to previous methods.     

基于CDN和P2P的分布式网络存储系统

把用户的文件分片后均衡存储在不同的分布式存储节点上,并利用虚拟目录服务器和基于P2P—DHT的目录服务器把文件元数据与文件数据片高效地对应起来,以提供高效目录服务,分布式存储节点以P2P方式工作以快速完成用户对文件数据的请求任务。分布式网络存储系统DNSS充分利用了CDN和P2P的技术优势,有较高的可用性、可靠性和可扩展性。DNSS已经在中国科学技术大学应用。     

一种细粒度均衡的新型分布式文件放置算法

为保证访问负载的均衡分布,分布式存储系统往往依赖访问热度信息进行文件放置。然而,访问热度信息在文件存入系统时刻并不可知,并且随时间不断变化,依赖访问热度信息的放置算法需要不断调整文件的存储位置,产生高昂的迁移成本。本文提出一种细粒度均衡的新型分布式文件放置算法。该算法利用文件访问热度同已创建时间之间的相关性,通过保证各节点所存储数据量在创建时间维度上的细粒度相似性,实现较好的访问负载均衡。该算法仅基于文件的创建时间属性,该属性在文件存入系统时刻属于已知信息并且不随时间变化。实验结果表明,相较于HDFS系统的随机放置算法,本文算法能够更好地实现访问负载的均衡分布,提高访问性能。     

Algorithms and performance of load-balancing with multiple hash functions in massive content distribution

We consider a two-tier content distribution system for distributing massive content, consisting of an infrastructure content distribution network (CDN) and a large number of ordinary clients. The nodes of the infrastructure network form a structured, distributed-hash-table-based (DHT) peer-to-peer (P2P) network. Each file is first placed in the CDN, and possibly, is replicated among the infrastructure nodes depending on its popularity. In such a system, it is particularly pressing to have proper load-balancing mechanisms to relieve server or network overload. The subject of the paper is on popularity-based file replication techniques within the CDN using multiple hash functions. Our strategy is to set aside a large number of hash functions. When the demand for a file exceeds the overall capacity of the current servers, a previously unused hash function is used to obtain a new node ID where the file will be replicated. The central problems are how to choose an unused hash function when replicating a file and how to choose a used hash function when requesting the file. Our solution to the file replication problem is to choose the unused hash function with the smallest index, and our solution to the file request problem is to choose a used hash function uniformly at random. Our main contribution is that we have developed a set of distributed, robust algorithms to implement the above solutions and we have evaluated their performance. In particular, we have analyzed a random binary search algorithm for file request and a random gap removal algorithm for failure recovery.     

Online Bicriteria Load Balancing Using Object Reallocation

We study the bicriteria load balancing problem on two independent parameters under the allowance of object reallocation. The scenario is a system of $M$ distributed file servers located in a cluster, and we propose three online approximate algorithms for balancing their loads and required storage spaces during document placement. The first algorithm is for heterogeneous servers. Each server has its individual tradeoff of load and storage space under the same rule of selection. The other two algorithms are for homogeneous servers. The second algorithm combines the idea of the first one and the best existing solution for homogeneous servers. Using document reallocation, we obtain a smooth tradeoff curve of the upper bounds of load and storage space. The last one bounds the load and storage space of each server by less than three times of their trivial lower bounds, respectively; and more importantly, for each server, the value of at least one parameter is far from its worst case. The time complexities of these three algorithms are $O(log M)$ plus the cost of document reallocation.     

Freenet中的密钥和索引机制综述

Freenet,又叫自由网,是一种分布式信息存储和搜索系统,设计定位于保障信息的私有性和有效性。系统的运行类似于一个具有位置无关性特征的分布式文件系统,这个文件系统由许多独立的计算机组成,这些计算机允许用户匿名进行文件的插入、存储和请求。在自由网中,发布的匿名性,文件的标识、发布、存储、请求都与自由网的密钥和索引机制有着相当密切的联系,该文对自由网中的密钥机制做了一定程度的分析、探索,以求能对自由网的运行机制的进一步研究和了解有所帮助。