Block I/O Scheduling on Storage Servers of Distributed File Systems

This paper presents a new scheme of I/O scheduling on storage servers of distributed/parallel file systems, for yielding better I/O performance. To this end, we first analyze read/write requests in the I/O queue of storage server (we name them block I/Os), by using our proposed technique of horizontal partition. Then, all block requests are supposed to be divided into multiple groups, on the basis of their offsets. This is to say, all requests related to the same chunk file will be grouped together, and then be satisfied within the same time slot between opening and closing the target chunk file on the storage server. As a result, the time resulted by completing block I/O requests can be significantly decreased, because of less file operations on the corresponding chunk files at the low-level file systems of server machines. Furthermore, we introduce an algorithm to rate a priority for each group of block I/O requests, and then the storage server dispatches groups of I/Os by following the priority order. Consequently, the applications having higher I/O priorities, e.g. they have less I/O operations and small size of involved data, can finish at a earlier time. We implement a prototype of this server-side scheduling in the PARTE file system, to demonstrate the feasibility and applicability of the proposed scheme. Experimental results show that the newly proposed scheme can achieve better I/O bandwidth and less I/O time, compared with the strategy of First Come First Served, as well as other server-side I/O scheduling approaches.     

Multicast Pull Scheduling: When Fairness Is Fine

We investigate server scheduling policies to minimize average user perceived latency in pull-based client-server systems (systems where multiple clients request data from a server) where the server answers requests on a multicast/ broadcast channel. We first show that there is no O(1) -competitive algorithm for this problem. We then give a method to convert any nonclairvoyant unicast scheduling algorithm A to nonclairvoyant multicast scheduling algorithm B . We show that if A works well, when jobs can have parallel and sequential phases, then B works well if it is given twice the resources. More formally, if A is an s -speed c -approximation unicast algorithm, then its counterpart algorithm B is a 2s -speed c -approximation multicast algorithm. It is already known [5] that Equi-partition, which devotes an equal amount of processing power to each job, is a (2 + ε) -speed O(1 + 1/ε) -approximation algorithm for unicast scheduling of such jobs. Hence, it follows that the algorithm {BEQUI}, which broadcasts all requested files at a rate proportional to the number of outstanding requests for that file, is a (4 + ε) -speed O(1 + 1/ε) -approximation algorithm. We give another algorithm BEQUI-EDF and show that BEQUI-EDF is also a (4 + ε) -speed O(1 + 1/ε) -approximation algorithm. However, BEQUI-EDF has the advantage that the maximum number of preemptions is linear in the number of requests, and the advantage that no preemptions occur if the data items have unit size.     

Disk scheduling in video editing systems

Modern video servers support both video-on-demand and nonlinear editing applications. Video-on-demand servers enable the user to view video clips or movies from a video database, while nonlinear editing systems enable the user to manipulate the content of the video database. Applications such as video and news editing systems require that the underlying storage server be able to concurrently record live broadcast information, modify prerecorded data, and broadcast an authored presentation. A multimedia storage server that efficiently supports such a diverse group of activities constitutes the focus of this study. A novel real-time disk scheduling algorithm is presented that treats both read and write requests in a homogeneous manner in order to ensure that their deadlines are met. Due to real-time demands of movie viewing, read requests have to be fulfilled within certain deadlines; otherwise, they are considered lost. Since the data to be written into disk is stored in main memory buffers, write requests can be postponed until critical read requests are processed. However, write requests still have to be processed within reasonable delays and without the possibility of indefinite postponement. This is due to the physical constraint of the limited size of the main memory write buffers. The new algorithm schedules both read and write requests appropriately, to minimize the amount of disk reads that do not meet their presentation deadlines, and to avoid indefinite postponement and large buffer sizes in the case of disk writes. Simulation results demonstrate that the proposed algorithm offers low violations of read deadlines, reduces waiting time for lower priority disk requests, and improves the throughput of the storage server by enhancing the utilization of available disk bandwidth     

Optimized dependent file fetch middleware in transparent computing platform

A middleware is proposed to optimize file fetch process in transparent computing (TC) platform. A single TC server will receive file requests of large scale distributed operating systems, applications or user data from multiple clients. In consideration of limited size of server’s memory and the dependency among files, this work proposes a middleware to provide a file fetch sequence satisfying: (1) each client, upon receiving any file, is able to directly load it without waiting for pre-required files (i.e. “receive and load”); and (2) the server is able to achieve optimization in reducing overall file fetch time cost. The paper firstly addresses the features of valid file fetch sequence generating problem in the middleware. The method solves the concurrency control problem when the file fetch is required for the multiple clients. Then it explores the methods to determine time cost for file fetch sequence. Based on the established model, we propose a heuristic and greedy (HG) algorithm. According to the simulation results, we conclude that HG algorithm is able to reduce overall file fetch time roughly by 50% in the best cases compared with the time cost of traditional approaches.     

Information delivery through broadcasting in satellite communication networks

Satellite broadcast is an important candidate for large-scale multimedia information distribution due to the inherent wide-range multicasting capability of satellites and the asymmetry of satellite communications (high bandwidth downlink, limited bandwidth uplink) that matches nicely the information flow asymmetry in multimedia applications. We consider a data broadcasting model that is encountered in most asymmetric satellite communication environments. The problem of scheduling the data broadcast such that average response time experienced by the users is low is considered. In a push-based system, where the users cannot place requests directly to the server and the broadcast schedule should be determined based solely on the access probabilities, we formulate a deterministic dynamic optimization problem, the solution of which provides the optimal broadcast schedule. Properties of the optimal solution are obtained and then we propose a suboptimal dynamic policy which achieves average response time close to the lower bound. In a pull-based system where the users may place requests about information items directly to the server, the scheduling can be based on the number of pending requests for each item. Suboptimal policies with good performance are obtained in this case as well. If a user has local memory, it can alleviate its access latency by selectively prefetching the items from the broadcast and storing them in the memory. A good memory management strategy can substantially reduce the user's access latency. An optimal memory management policy is identified, that minimizes the expected aggregate latency. Memory update strategies with limited look-ahead are presented as implementable approximations of the optimal policy as well. We also consider the problem of joint broadcast scheduling and user's cache management and propos a joint optimization scheme which can achieve the performance up to 40% better than the existing non-joint approach.     

一种资源优化的双最小均衡Web集群区分服务调度算法

在一种新的Web集群体系结构的基础上,提出了一种资源优化的双最小均衡区分服务调度算法：首先在前端调度器按资源均衡度将Web请求分配到各后台服务器．然后将Web请求的优先级与资源均衡度两个特征参数结合起来,综合设计后台服务器的Web请求调度顺序,为了评估该算法的性能,进行了大量的模拟实验．在与其他著名调度策略如分离式调度的对比结果显示：双最小均衡调度算法使Web请求的效率提高了11％,同时很好地实现了区分服务．证实了资源优化调度策略具有一定的普遍意义．     

Queueing models for a single server LAN with processor sharing disciplines

A local area network (LAN) is a collection of autonomous workstations interconnected by a communication network. A key component of a local network is the file server which stores programs and data and makes them available to the workstations as needed. In practice, a workstation requests a large portion of the file (type 1 request) when a new application is launched. Following this, the workstation requests additional portions of the file as needed (type 2 requests). Clearly, the response time to these requests will depend strongly on the file server scheduling policy to service the two types of incoming requests. In an earlier paper [12] we studied this system when type 2 requests havepreemptive and non-preemptive priority over type 1 requests. From a practical point of view, it is more realistic to describe the file transfer by around-robin scheduling discipline. In this paper we study LANs where the file server followsprocessor sharingdiscipline, which is a limiting case of theround-robin discipline. Assuming that all relevant intervals are exponentially distributed we develop algorithms to analyze the performance of the system. Illustrative examples are presented to study the system behavior under various load conditions. The computational approach presented in this paper helps in resolving some of the analytical difficulties associated with the analysis ofprocessor sharing disciplines.     

一种松竞争性缓存快速存取算法

在文件缓存调度中,每个文件都有固定的大小和被存取的消耗,为了响应对文件操作的一系列请求,把缓存中所有文件的大小维持在一个特定的k值之内,从而最小化文件存取的总消耗。给出一个简单明确的快速存取算法,该算法总结了许多有名的内存分页策略和加权缓存策略,证明了对于大多数k的选择,存取消耗可以忽略不计或者是最佳值的恒定倍数(与k值无关)。从而证明了在线分页算法的竞争比可视为一个常数。     

NP-Hardness of Broadcast Scheduling and Inapproximability of Single-Source Unsplittable Min-Cost Flow

We consider the version of broadcast scheduling where a server can transmit W messages of a given set at each time-step, answering previously made requests for these messages. The goal is to minimize the average response time (ART) if the amount of requests is known in advance for each time-step and message. We prove that this problem is NP-hard, thus answering an open question stated by Kalyanasundaram, Pruhs and Velauthapillai (Proceedings of ESA 2000, LNCS 1879, 2000, pp. 290–301). Furthermore, we present an approximation algorithm that is allowed to send several messages at once. Using six channels for transmissions, the algorithm achieves an ART that is at least as good as the optimal solution using one channel. From the NP-hardness of broadcast scheduling we derive a new inapproximability result of (2 − ε, 1) for the (congestion, cost) bicriteria version of the single source unsplittable min-cost flow problem, for arbitrary ε > 0. The result holds even in the often considered case where the maximum demand is less than or equal to the minimum edge capacity (d _max ≤ u _min), a case for which an algorithm with ratio (3, 1) was presented by Skutella.     

Dynamic Scheduling of Multimedia Documents in a Single Server Multiple Clients Environment

In a typical single server and multiple client distributed multimedia system, clients may send sporadic requests to the server for certain multimedia documents. The requests must be served with a fast response time and with the required quality of service guarantee. This requires the server to determine the transmission schedule of each multimedia stream while ensuring necessary inter- and intrastream synchronizations. There are two major drawbacks in the existing scheduling algorithms. First, it is assumed that all channels are available at the beginning of the scheduling, but in reality, requests arrive when others are in service; second, the cost of the scheduling itself is usually ignored. In general a feasible scheduling algorithm should have the following features: (1) the schedule must be generated in real time, (2) it should have small scheduling cost, and (3) it must be capable of handling multiple requests from multiple clients. In this paper, we propose two dynamic scheduling algorithms whose worst time complexity isO(n log nm+nm), wherenis the total number of data units in a retrieved multimedia document andmdenotes the number of available channels. The salient feature of the proposed algorithms is their inherent dynamic nature which can adjust the scheduling times for each individual request according to the slack time between consecutive requests. If the slack time between two requests is large, the scheduler can run longer in an attempt to find a better solution. This reduces the response time while maintaining a good quality of presentation. Through both simulation and analysis, we evaluate our algorithms and demonstrate their applicability in a realistic environment.     

Scheduling broadcasts with deadlines

We investigate the problem of scheduling broadcasts in data delivering systems via broadcast, where a number of requests from several clients can be simultaneously satisfied by one broadcast of a server. Most of prior work has focused on minimizing the total flow time of requests. It assumes that once a request arrives, it will be held until satisfied. In this paper, we are concerned with the situation that clients may leave the system if their requests are still unsatisfied after waiting for some time, that is, each request has a deadline. The problem of maximizing the throughput, for example, the total number of satisfied requests, is developed, and there are given online algorithms achieving constant competitive ratios.     

Multicast Pull Scheduling: When Fairness Is Fine

Abstract. We investigate server scheduling policies to minimize average user perceived latency in pull-based client-server systems (systems where multiple clients request data from a server) where the server answers requests on a multicast/ broadcast channel. We first show that there is no O(1) -competitive algorithm for this problem. We then give a method to convert any nonclairvoyant unicast scheduling algorithm A to nonclairvoyant multicast scheduling algorithm B . We show that if A works well, when jobs can have parallel and sequential phases, then B works well if it is given twice the resources. More formally, if A is an s -speed c -approximation unicast algorithm, then its counterpart algorithm B is a 2s -speed c -approximation multicast algorithm. It is already known [5] that Equi-partition, which devotes an equal amount of processing power to each job, is a (2 + ε) -speed O(1 + 1/ε) -approximation algorithm for unicast scheduling of such jobs. Hence, it follows that the algorithm {BEQUI}, which broadcasts all requested files at a rate proportional to the number of outstanding requests for that file, is a (4 + ε) -speed O(1 + 1/ε) -approximation algorithm. We give another algorithm BEQUI-EDF and show that BEQUI-EDF is also a (4 + ε) -speed O(1 + 1/ε) -approximation algorithm. However, BEQUI-EDF has the advantage that the maximum number of preemptions is linear in the number of requests, and the advantage that no preemptions occur if the data items have unit size.     

Optimal Broadcasting of Two Files over an Asymmetric Channel

We study the problem of scheduling files over a broadcast channel in an asymmetric environment. The goal is to minimize the mean response time for clients who access the broadcast channel. Asymmetric channels have gained a lot of attention because they are used to model wireless ccxnmunication, teletext systems, and web caching in satellite systems. This paper addresses the two-files case. We design a simple algorithm that defines the optimal schedule given the demand probability for each file. Our solution is extended to include other important factors, such as dependencies between files, variable-length files, and other extensions, such as different priorities for the clients. For all the above extensions, we prove the surprising result that there exists a simple optimal schedule. Such a schedule is composed of a repeated pattern of AA…AB, where A is the more “popular” file and B is the less “popular” file.     

adPD：一种速度自适应的动态并行下载技术

本文在介绍了现有的并行下载算法的基础上提出了一种新的速度自适应的动态并行下载机制-adPD。adPD通过为速度不同的连接动态分配大小不同的下载任务,可以很好地适应传输连接速度的变化,做到按速度比例分配下载任务量,充分利用带宽。同时,通过划分大小不固定的文件分块,adPD还可以尽可能地减少发送数据请求的数量,缩短请求等待的空闲时间,在减轻提供服务的节点的负载的同时,提高了下载速度。最后,通过实验结果分析了adPD的实际性能,验证了adPD是一种高效的并行下载算法。     

基于请求的多信道多数据项广播调度算法*

通过改进多信道单数据项请求的数据广播调度算法中的两层调度策略,提出了信道分配+QEM的算法;实现了用多信道广播来广播多数据项的请求;通过减少数据访问冲突和信道跳转减少了访问时间。分析证明此方法是可行和有效的。     

On-demand broadcast for multiple-item requests in a multiple-channel environment

On-demand broadcast is an effective approach to disseminating data in mobile computing environments. Substantial efforts have been devoted to improving the scheduling efficiency of on-demand broadcast. Previous studies focused mainly on the case of scheduling single-item requests in single-channel environments. However, requesting multiple dependent data items is common in many advanced applications such as electronic stock trading and traffic information enquiry services. In addition, multi-channel architectures are widely deployed in data broadcast systems. In this work, we investigate the issues arising in scheduling multi-item requests in multi-channel on-demand broadcast environments. Two problems, namely, the request starvation problem and the bandwidth utilization problem are identified in existing algorithms. To tackle the observed problems, an innovative algorithm is proposed. Results from our simulation study demonstrate the superiority of the proposed algorithm.     

移动实时环境下一种改进的广播调度算法

在网络带宽不对称的移动实时环境中,数据广播是一种有效的数据访问方式。针对这种网络特性,分析了现今已经存在的某些广播调度算法。针对UFO算法,分别提出了SBS算法和CRS算法,它们从服务器、移动客户端两个方面进行了改进。两种算法可以根据给定的数据项访问概率分布,自动生成广播调度。通过理论分析和实验结果表明,该算法不会产生事务重启,并且可以有效减少数据的访问时间,使用户访问数据广播的平均等待时间最小。     

基于小文件的内存云存储优化策略

由于内存云RAMCloud采用日志段的方式存储数据,因此当大量小文件存储于RAMCloud集群时,每个小文件独占整个段,会产生较多的段内碎片,从而导致内存的有效利用率较低以及大量的内存空间浪费。为了解决这个问题,提出基于文件分类的RAMCloud小文件存储优化策略。该策略首先根据文件的相关特性将小文件分为结构相关文件、逻辑相关文件以及相互独立文件三类;然后在存储时对结构相关的文件使用文件合并算法,逻辑相关和相互独立的小文件则使用分组算法。实验结果表明:同未进行优化的RAMCloud存储策略相比,该策略能有效提高集群内存利用率。     

Dynamic quota-based admission control with sub-rating in multimedia servers

An admission control algorithm for a multimedia server is responsible for determining if a new request can be accepted without violating the Quality of Service (QoS) requirements of the existing requests in the system. A novel quota-based admission control algorithm with sub-rating for two priority classes of requests is proposed in this study. The server capacity is divided into three partitions based on the quota values: one for each class of requests and one common pool shared by two classes of requests. Reward and penalty are adopted in the proposed system model. High-priority requests are associated with higher values of reward as well as penalty than low-priority ones. Given the characteristics of the system workload, the proposed algorithm finds the best partitions, optimizing the system performance based on the objective function of the total reward minus the total penalty. The sub-rating mechanism will reduce the QoS requirements of several low- priority clients, by cutting out a small fraction of the assigned server capacity, to accept a new high- priority client and to achieve a higher net earning value. A stochastic Petri-Net model is used to find the optimal quota values and two approximation approaches are developed to find sub-optimal settings. The experiment results show that the proposed algorithm performs better than one without sub-rating mechanism, and that the sub-optimal solutions found by the proposed approximation approaches are very close to optimal ones. The approximation approaches enable the algorithm to dynamically adjust the quota values, based on the characteristics of the system workload, to achieve higher system performance.     

Comparing files using structural entropy

One of the main trends in the modern anti-virus industry is the development of algorithms that help estimate the similarity of files. Since malware writers tend to use increasingly complex techniques to protect their code such as obfuscation and polymorphism, anti-virus software vendors face problems of the increasing difficulty of file scanning, the considerable growth of anti-virus databases, and file storages overgrowth. For solving such problems, a static analysis of files appears to be of some interest. Its use helps determine those file characteristics that are necessary for their comparison without executing malware samples within a protected environment. The solution provided in this article is based on the assumption that different samples of the same malicious program have a similar order of code and data areas. Each such file area may be characterized not only by its length, but also by its homogeneity. In other words, the file may be characterized by the complexity of its data order. Our approach consists of using wavelet analysis for the segmentation of files into segments of different entropy levels and using edit distance between sequence segments to determine the similarity of the files. The proposed solution has a number of advantages that help detect malicious programs efficiently on personal computers. First, this comparison does not take into account the functionality of analysed files and is based solely on determining the similarity in code and data area positions which makes the algorithm effective against many ways of protecting executable code. On the other hand, such a comparison may result in false alarms. Therefore, our solution is useful as a preliminary test that triggers the running of additional checks. Second, the method is relatively easy to implement and does not require code disassembly or emulation. And, third, the method makes the malicious file record compact which is significant when compiling anti-virus databases.