Scheduling multi‐component applications with mobile agents in heterogeneous distributed systems

In grid computing environment, several classes of multi‐component applications exist. These types of applications may often require additional resources of different types that go beyond what is available in any of the sites making up the grid resource composition. The heterogeneity nature of both the user application and the computing environment makes this a challenging problem. However, the current off‐the‐shelf scheduling software can hardly cope with these diversities in distributed computing application frameworks. Therefore, there is the need for an adequate scheduling system that would grant simultaneous or coordinated access to application of multi‐component nature that requires resources of possibly multiple types, in multiple locations, managed by different resource providers. The main focus of this paper is to develop a mobile agent scheduling model that addresses the aforementioned challenge. A scheduling policy that pertains to job scheduling and resource allocation is proposed. The scheduling policy treats different multi‐component applications requiring diverse heterogeneous resources fairly. The policy is used by mobile agents to schedule user applications and to also find available and suitable distributed resource that are capable of executing user application at a very minimal time. Copyright © 2015 John Wiley & Sons, Ltd.     

ServiceSs: An Interoperable Programming Framework for the Cloud

The rise of virtualized and distributed infrastructures has led to new challenges to accomplish the effective use of compute resources through the design and orchestration of distributed applications. As legacy, monolithic applications are replaced with service-oriented applications, questions arise about the steps to be taken in order to maximize the usefulness of the infrastructures and to provide users with tools for the development and execution of distributed applications. One of the issues to be solved is the existence of multiple cloud solutions that are not interoperable, which forces the user to be locked to a specific provider or to continuously adapt applications. With the objective of simplifying the programmers challenges, ServiceSs provides a straightforward programming model and an execution framework that helps on abstracting applications from the actual execution environment. This paper presents how ServiceSs transparently interoperates with multiple providers implementing the appropriate interfaces to execute scientific applications on federated clouds.     

A Methodology for Structured Database Decomposition

We present a methodology for structured database decomposition based on the relational data model. It is argued that in the distributed database environment, structured database decomposition is attractive both for efficiency and for database security considerations. Techniques for parallel processing and hashed access of structurally decomposed database are presented. Techniques for structured database decomposition to support multiple user views are also described. Structured database decomposition is most advantageous in a query only database environment with stable user views, although dynamic updates can also be handled using techniques described in this paper.     

Middleware infrastructure for parallel and distributed programming models in heterogeneous systems

We introduce a middleware infrastructure that provides software services for developing and deploying high-performance parallel programming models and distributed applications on clusters and networked heterogeneous systems. This middleware infrastructure utilizes distributed agents residing on the participating machines and communicating with one another to perform the required functions. An intensive study of the parallel programming models in Java has helped identify the common requirements for a runtime support environment, which we used to define the middleware functionality. A Java-based prototype, based on this architecture, has been developed along with a Java object-passing interface (JOPI) class library. Since this system is written completely in Java, it is portable and allows executing programs in parallel across multiple heterogeneous platforms. With the middleware infrastructure, users need not deal with the mechanisms of deploying and loading user classes on the heterogeneous system. Moreover, details of scheduling, controlling, monitoring, and executing user jobs are hidden, while the management of system resources is made transparent to the user. Such uniform services are essential for facilitating the development and deployment of scalable high-performance Java applications on clusters and heterogeneous systems. An initial deployment of a parallel Java programming model over a heterogeneous, distributed system shows good performance results. In addition, a framework for the agents' startup mechanism and organization is introduced to provide scalable deployment and communication among the agents.     

Fog computing for next-generation Internet of Things: Fundamental,state-of-the-art and research challenges

In recent times, the Internet of Things (IoT) applications, including smart transportation, smart healthcare, smart grid, smart city, etc. generate a large volume of real-time data for decision making. In the past decades, real-time sensory data have been offloaded to centralized cloud servers for data analysis through a reliable communication channel. However, due to the long communication distance between end-users and centralized cloud servers, the chances of increasing network congestion, data loss, latency, and energy consumption are getting significantly higher. To address the challenges mentioned above, fog computing emerges in a distributed environment that extends the computation and storage facilities at the edge of the network. Compared to centralized cloud infrastructure, a distributed fog framework can support delay-sensitive IoT applications with minimum latency and energy consumption while analyzing the data using a set of resource-constraint fog/edge devices. Thus our survey covers the layered IoT architecture, evaluation metrics, and applications aspects of fog computing and its progress in the last four years. Furthermore, the layered architecture of the standard fog framework and different state-of-the-art techniques for utilizing computing resources of fog networks have been covered in this study. Moreover, we included an IoT use case scenario to demonstrate the fog data offloading and resource provisioning example in heterogeneous vehicular fog networks. Finally, we examine various challenges and potential solutions to establish interoperable communication and computation for next-generation IoT applications in fog networks.     

分布式系统下的DAG任务调度研究综述

近年来随着网格、云计算工作流等分布式计算技术的发展,关于DAG(有向无环图)模型任务在分布式系统环境下的调度问题逐渐成为备受关注的研究热点。根据最新研究进展,对分布式系统下的DAG任务调度问题和有关技术进行了研究与讨论,主要包括四个方面:系统地描述了分布式系统和异构分布式系统的有关概念,异构分布式系统下的DAG任务调度问题、调度模型及其典型应用;对现有分布式系统下DAG任务调度的研究按照不同的方式进行了分类;探讨了多DAG共享异构分布式资源调度的研究现状;讨论了目前多DAG共享异构分布式资源调度研究存在的问题和未来可能的研究方向。     

Clustering semantically heterogeneous distributed aggregate databases

Databases developed independently in a common open distributed environment may be heterogeneous with respect to both data schema and the embedded semantics. Managing schema and semantic heterogeneities brings considerable challenges to learning from distributed data and to support applications involving cooperation between different organisations. In this paper, we are concerned mainly with heterogeneous databases that hold aggregates on a set of attributes, which are often the result of materialised views of native large-scale distributed databases. A model-based clustering algorithm is proposed to construct a mixture model where each component corresponds to a cluster which is used to capture the contextual heterogeneity among databases from different populations. Schema heterogeneity, which can be recast as incomplete information, is handled within the clustering process using Expectation-Maximisation estimation and integration is carried out within a clustering iteration. Our proposed algorithm resolves the schema heterogeneity as part of the clustering process, thus avoiding transformation of the data into a unified schema. Results of algorithm evaluation on classification, scalability and reliability, using both real and synthetic data, demonstrate that our algorithm can achieve good performance by incorporating all of the information from available heterogeneous data. Our clustering approach has great potential for scalable knowledge discovery from semantically heterogeneous databases and for applications in an open distributed environment, such as the Semantic Web.     

Secure interoperable digital content distribution mechanisms in a multi-domain architecture

Current DRM systems use the Authorized Domain concept to allow sharing of DRM-enabled multimedia contents across multiple devices. However, some devices in an authorized domain may support only a limited number of DRM systems of the content providers due to their heterogeneous capabilities. Lack of interoperability among DRM systems enforces these devices to stick to a common DRM system which restricts the sharing of different DRM-enabled multimedia contents among them. Most of the current solutions use a translation entity to provide interoperability among different DRM standards with a trust assumption over that entity. This assumption may not assure the content providers that their contents and licenses will be translated and distributed in a secure and legal way. In this paper, we propose a secure interoperable content distribution mechanism for commercial and user generated contents among multiple authorized domains without any trust assumption on the translation entity.     

Semantic interoperability in building design: Methods and tools

Semantic interoperability is a crucial element to make building information models understandable and model data sharable across multiple design disciplines and heterogeneous computer systems. This paper presents a new approach and its software implementation for the development of building design objects with semantics of interoperable information to support semantic interoperability in building designs. The novelty of the approach includes its incorporation of building design domain ontology, object-based CAD information modeling, and interoperability standard to make building information models and model data semantically interoperable. A set of methods are proposed to address the issues of object-based building information representation compliant with the Industrial Foundation Classes (IFC); extension of IFC models with the supplementary information; and semantic annotation of the interoperable and extensible information sets. The prototype implementation of these methods provides a set of Web-enabled software tools for effectively generating, managing, and reusing the semantically interoperable building objects in design applications of architectural CAD, structural analysis, and building code conformance checking.     

Independent checkpointing in a heterogeneous grid environment

The EU-funded XtreemOS project implements an open-source grid operating system based on Linux. In order to provide fault tolerance and migration for grid applications, it integrates a distributed grid-checkpointing service called XtreemGCP. This service is designed to support various checkpointing protocols and different checkpointer packages (e.g. BLCR, LinuxSSI, OpenVZ, etc.) in a transparent manner through a uniform checkpointer interface. In this paper, we present the integration of a backward error recovery protocol based on independent checkpointing into the XtreemGCP service. The solution we propose is not checkpointer bound and thus can be transparently used on top of any checkpointer package.To evaluate the prototype we run it within a heterogeneous environment composed of single-PC nodes and a Single System Image (SSI) cluster. The experimental results demonstrate the capability of the XtreemGCP service to integrate different checkpointing protocols and independently checkpoint a distributed application within a heterogeneous grid environment. Moreover, the performance evaluation also shows that our solution outperforms the existing coordinated checkpointing protocol in terms of scalability.     

Digging Deep into the Data Mine with DataMiningGrid

The growing computerization in modern knowledge and technology sectors is generating huge volumes of electronically stored data. Data mining technology is often employed to make sense of these data. However, as modern data mining applications increase in complexity, so do their demands for resources. Grid computing is one of several emerging networked computing paradigms promising to meet the requirements of heterogeneous, large-scale and distributed data mining applications. Despite this promise, there are still too many issues to be resolved before grid technology is commonly applied to large-scale data mining tasks. To address some of these issues, we developed the DataMiningGrid system, which principally differs from similar systems by its ability to integrate a diverse set of programs and application scenarios within a single framework. The system's key features include high performance and scalability, sophisticated support for relevant standards, different user types, and flexible extensibility. The software is available as open source.     

Optional and responsive fine-grain locking in Internet-based collaborative systems

Locking is a standard technique used in distributed computing and database systems to ensure data integrity by prohibiting concurrent conflicting updates on shared data objects. Internet-based collaborative systems are a special class of distributed applications which support human-to-human interaction and collaboration over the Internet. In this paper, a novel optional and responsive fine-grain locking scheme is proposed for consistency maintenance in Internet-based collaborative editors. In the proposed scheme, locking is made optional in the sense that a user may update any part of the document without necessarily requesting a lock. In the face of high communication latency in the Internet environment, responsive locking is achieved by granting the permit to the user for updating the data region immediately after issuing a locking request. Moreover, multiple fine-grain locks can be placed on different regions inside a document to allow concurrent and mutually exclusive editing on the same document. Protocols and algorithms for locking conflict resolution and consistency maintenance are devised to address special technical issues involved in optional and responsive fine-grain locking. The proposed locking scheme and supporting techniques were implemented in an Internet-based collaborative editor to demonstrate its feasibility and usability.     

基于网格的物流资源挖掘模型的研究

分布在因特网上的物流资源具有地理分散和职权自治的特性,资源结构和接口难以统一。该文以网格、Agent和增量挖掘技术为基础,提出了不通过资源整合就能够实现全局信息挖掘的方法。分析了基于Web的物流资源网格系统,将其划分为物流域的集合实施分级管理。提出了新的面向网格的信息挖掘模型并设计了域内动态资源挖掘算法和域间请求式资源挖掘算法。该模型解决了不同物流系统之间的信息挖掘难题,算法中引入的增量挖掘技术提高了域间资源挖掘效率。     

基于知识网格的数据挖掘

工业、科学、商务等领域的数据通常分布在不同的地方,需要在不同的地点对其进行分布式维护。只有使用计算功能超强的分布式、并行处理系统才能分析这些领域所产生的超大规模数据集。网格为分布式知识发现应用中的计算提供了有效支持。为了在网格上进行数据挖掘的开发,本文提供了一个称之为知识网格的系统,讨论如何应用知识网格设计实施数据挖掘应用,并说明如何搜索网格资源、编制软件和数据组件,以及数据挖掘应用在网格上的执行过程。     

Optimizing cyclic join view maintenance over distributed data sources

Materialized views defined over distributed data sources are critical for many applications to ensure efficient access, reliable performance, and high availability. Materialized views need to be maintained upon source updates since stale view extents may not serve well or may even mislead user applications. Thus, view maintenance performance is one of the keys to the success of these applications. In this work, we investigate two maintenance strategies, extended batching and view graph transformation, for maintaining general join views where join conditions may exist between any pairs of data sources possibly with cycles. Many choices are available for maintaining cyclic join views. We thus propose a cost-driven view maintenance framework which generates optimized maintenance plans tuned to the environmental settings. The proposed framework has been implemented in the TxnWrap system. Experimental studies illustrate that our proposed optimization techniques significantly improve the view maintenance performance in a distributed environment.     

基于知识网格的分布式数据挖掘

科学和工商业应用需要分析分布在各异构站点的海量数据。这就需要合适的分布式并行系统来存储和管理数据。网格为分布式数据挖掘和知识发现提供了有效的计算支持。文中在讨论知识网格体系结构的基础上，利用可视化网格应用环境VEGA实现了基于网格的分布式数据挖掘过程。     

Task scheduling,resource provisioning,and load balancing on scientific workflows using parallel SARSA reinforcement learning agents and genetic algorithm

The Journal of Supercomputing - Cloud computing is one of the most popular distributed environments, in which, multiple powerful and heterogeneous resources are used by different user applications....     

dOpenCL: Towards uniform programming of distributed heterogeneous multi-/many-core systems

Modern computer systems become increasingly distributed and heterogeneous by comprising multi-core CPUs, GPUs, and other accelerators. Current programming approaches for such systems usually require the application developer to use a combination of several programming models (e.g., MPI with OpenCL or CUDA) in order to exploit the system’s full performance potential. In this paper, we present dOpenCL (distributed OpenCL)—a uniform approach to programming distributed heterogeneous systems with accelerators. dOpenCL allows the user to run unmodified existing OpenCL applications in a heterogeneous distributed environment. We describe the challenges of implementing the OpenCL programming model for distributed systems, as well as its extension for running multiple applications concurrently. Using several example applications, we compare the performance of dOpenCL with MPI + OpenCL and standard OpenCL implementations.     

基于计算经济的网格资源管理模型研究

由于网格中的资源具有分布性、自治性、异构性和动态性,对网格资源进行高效的管理是一个具有挑战性的问题.提出把移动Agent和Agent技术、计算经济机制引入网格资源管理.自治Agent利用双向拍卖经济杠杆的调节,以用户的QoS所驱动,对资源的使用进行竞价拍卖,使网格的资源管理能够自动适应环境的变化.提出的基于计算经济的网格资源管理模型DynamicSA,实现了以用户的QoS驱动,对网格资源的有效管理具有一定的理论意义.     

基于语义的数据网格服务

探讨一种基于Wrapper—Mediator结构的支持语义的数据网格服务机制。首先探讨这种机制的总体结构,包括利用基于本体的语义来封装各个异质数据源,并通过Mediator结构来实现对各种异质数据源的支持,以及通过OTX算法来实现虚拟数据源VDS来支持OGSA-DAI接口标准。然后讨论其数据访问机制,即扩展XML代数来支持基于语义的数据访问。最后介绍一种支持语义的网格通信机制及其通信原语。