Workflow-Based Data Parallel Applications on the EGEE Production Grid Infrastructure

Setting up and deploying complex applications on a Grid infrastructure is still challenging and the programming models are rapidly evolving. Efficiently exploiting Grid parallelism is often not straight forward. In this paper, we report on the techniques used for deploying applications on the EGEE production Grid through four experiments coming from completely different scientific areas: nuclear fusion, astrophysics and medical imaging. These applications have in common the need for manipulating huge amounts of data and all are computationally intensive. All the cases studied show that the deployment of data intensive applications require the development of more or less elaborated application-level workload management systems on top of the gLite middleware to efficiently exploit the EGEE Grid resources. In particular, the adoption of high level workflow management systems eases the integration of large scale applications while exploiting Grid parallelism transparently. Different approaches for scientific workflow management are discussed. The MOTEUR workflow manager strategy to efficiently deal with complex data flows is more particularly detailed. Without requiring specific application development, it leads to very significant speed-ups.     

Practical Experience from Porting and Executing the Wien2k Application on the EGEE Production Grid Infrastructure

While the Grid promises to deliver a large number of computation nodes to a user, this computation power is not usable without the proper adaption of the application for the Grid. In this paper, we describe the methods used to port and execute a particular application, Wien2k, on the EGEE production Grid. First, the process of porting the application is described. Then, we investigate the measures necessary to execute the application in this production Grid environment efficiently. Although the focus is on this special application, we describe generic methods which can be applied to all applications. We specifically address: Creating a workflow from an application and mapping this workflow to a Grid workflow using the activity attraction pattern. We discuss workflow engines which support cycles in their application workflow. We investigate naïve and worker scheduling techniques. A short introduction into licensing on the Grid is given. Optimisation techniques such as deployment re-use are discussed. Different data transfer mechanisms, centralised data transfer, data re-use, storage element data transfer, and peer-to-peer data transfer are compared. The paper is concluded with suggestions for further workflow porting.     

Statistical Analysis and Modeling of Jobs in a Grid Environment

The existence of good probabilistic models for the job arrival process and the delay components introduced at different stages of job processing in a Grid environment is important for the improved understanding of the Grid computing concept. In this study, we present a thorough analysis of the job arrival process in the EGEE infrastructure and of the time durations a job spends at different states in the EGEE environment. We define four delay components of the total job delay and model each component separately. We observe that the job inter-arrival times at the Grid level can be adequately modelled by a rounded exponential distribution, while the total job delay (from the time it is generated until the time it completes execution) is dominated by the computing element’s register and queuing times and the worker node’s execution times. Further, we evaluate the efficiency of the EGEE environment by comparing the job total delay performance with that of a hypothetical ideal super-cluster and conclude that we would obtain similar performance if we submitted the same workload to a super-cluster of size equal to 34% of the total average number of CPUs participating in the EGEE infrastructure. We also analyze the job inter-arrival times, the CE’s queuing times, the WN’s execution times, and the data sizes exchanged at the kallisto.hellasgrid.gr cluster, which is node in the EGEE infrastructure. In contrast to the Grid level, we find that at the cluster level the job arrival process exhibits self-similarity/long-range dependence. Finally, we propose simple and intuitive models for the job arrival process and the execution times at the cluster level.     

Resources and Services of the EGEE Production Infrastructure

This paper presents the pan-European EGEE Grid focusing on aspects such as production infrastructure, the management tools and the operational services offered. Usage statistics and the provided Quality of Service are analysed to assess the maturity level, the current penetration of Grid technologies in Europe and the current expansion trends. Being EGEE a large distributed infrastructure, operations are a joint effort of different regional centres with central coordination. EGEE operations rely on a common and agreed set of procedures, policies and interfaces, which are the foundation of operational services such as middleware deployment, Grid oversight, accounting, operational security management and support. A transition is in place to lead EGEE to a more sustainable approach based on a set of integrated National Grid Initiatives. With the support of the EGI-InSPIRE project the EGEE e-infrastructure and its services will migrate into a new governance model for the future sustainability of Grids in Europe.     

利用ASP.NET的Web数据网格控件实现远程批量提交的方法

该文以班级德育量化考评数据批量提交为实例,重点阐述了利用Web数据网格控件GridView实现基于网页的批量更新提交的方法。     

Kubernetes集群上深度学习负载优化

人工智能技术的快速发展和在云原生上部署应用高效等优点让越来越多的开发者和互联网企业将人工智能应用部署在Kubernetes集群上,但Kubernetes并不是主要针对深度学习而设计,对深度学习这个特定领域需要做定制优化.本文针对具有一定规模的Kubernetes集群上部署深度学习负载的场景,设计和实现了一系列优化方案,主要从深度学习所要求的数据处理、graphics processing unit (GPU)计算、分布式训练等几个方面进行优化,本文提出的优化方案覆盖了数据处理、计算等方面,这些技术极大简化人工智能负载在规模化云原生平台上的部署难度和提高运行效率,同时从实践上来看也验证了以上技术对人工智能应用有着显著的提升作用.     

Availability Modeling and Cost Optimization for the Grid Resource Management System

Grid computing is a recently developed technique for complex systems with large-scale resource sharing, wide-area communication, and multi-institutional collaboration. Although the development tools and techniques for the grid have been extensively investigated, the availability of the grid resource management system (RMS) has not been comprehensively studied. In order to contribute to this lacking but important field, this paper first models the grid RMS availability by considering both the failures of resource management (RM) servers and the length limitation of request queues. A hierarchical Markov reward model is implemented to evaluate the grid RMS availability. Based on the availability model, an optimization problem for designing the grid RMS is studied in order to minimize the cost by determining the best number of RM servers. Then, the sensitivity analysis is conducted, and a dynamic switching scheduling method is further presented based on the sensitivity analysis.     

Economical Optimization of Grid Power Factor Using Predictive Data

We present an electrical grid optimization method for economical benefit. After simplifying an IEEE feeder diagram, we build a compact smart grid system including a photovoltaic-inverter system, a shunt capacitor, an on-load tapchanger (OLTC) and transmission lines. The system power factor (PF) regulation and reactive power dispatching are indispensable to improve power quality. Our control method uses predictive weather and load data to decide engaging or tripping the shunt capacitor, or reactive power injection by the photovoltaic-inverter system, ultimately to keep the system PF in a good range. From the perspective of economics, the economical model is considered as a decision maker in our predictive data control method. Capacitor-only control strategy is a common photovoltaic (PV) regulation method, which is treated as a baseline case. Simulations with GridLAB-D on profiled loads and residential loads have been carried out. The comparison results with baseline control strategy and our predictive data control method show the appreciable economical benefit of our method.      

基于Web的网格作业提交系统的研究与实现

作业管理是用户使用网格服务和网格资源的接口.本文提出了通用的作业管理系统框图和基于Web的作业提交模型,并详细分析了作业提交的过程,时用户管理、提交服务和任务处理等关键问题的实现做了阐述.     

断层地质体的三维建模技术研究

对复杂地质体中的断层三维建模技术进行研究与讨论。分析断层三维数据来源给出数据空间模型;系统地对‘已有的断层三维建模的技术进行分析比较,在此基础上提出一种使用TIN和三棱柱混合数据模型的断层地质体建模技术,从而有效地解决了断层三维可视化问题。     

整理车间多Agent生产管理系统的建模与过程优化

借助多Agent技术,提出了一种基于多Agent的生产管理结构模型,对用户实际需求和结构模型的合理性进行了分析.在此基础上,开发了一个柔性、动态、高效的生产管理系统,对车间目前的生产管理模式进行了优化.同时,对多个体Agent的主要功能、工作原理和系统建模过程进行了介绍.实际应用表明,多Agent的生产管理系统,解决了...     

巧用VB 的ADO处理各类数据库故障解析

操作系统从Win7 开始变成了64 位系统,导致VB下基本的界面操作都变得很卡,原先提供的众多命令、函数 等不能正常运行。幸好快捷而且强大的数据库连接工具ADO依然正常,但操作系统、数据库版本、链接库版本、数据库信息量 都会对数据产生影响,教程的理论分析与实际情况往往存在差异,对此本文主要分析笔者在编写程序时遇到的实际问题、困惑 以及解决方法。     

粒子群算法在电网无功优化中的应用研究

为解决目前电网系统无功优化潮流计算中存在的问题,如计算量大,计算结果中的各节点电压值可能导致无功电源出力接近极限值,并可能与系统电压安全发生冲突,发电机出力越限等。本文采用带罚函数、学习因子和惯性权重的改进粒子群算法,通过模拟编程,求解了在给定约束条件情况下,两个典型系统（5节点典型系统 和39节点典型系统）的无功优化潮流计算问题。通过计算结果分析比较,总结出了在无功优化计算中,如何对电网中的约束条件进行处理,以及如何设置粒子群算法中的相关参数和范围。并讨论了电网的约束条件对无功优化结果的影响,给出了粒子群算法中罚函数、惯性权重及学习因子等参数的设置原则以及对算法收敛性的影响,并对算法的改进进行了展望。     

基于人工鱼群算法的高级综合生产计划优化研究

充分考虑了多产品多阶段多分销商企业生产过程各种影响因素及其相互关系后，构造了它的计划优化模型。在此基础上，应用人工鱼群算法，对该模型进行研究，并在计算机上得以仿真实现。仿真结果表明，该算法不受问题规模的限制，对于解决大规模复杂的问题更显示出其优越性，为企业生产计划优化提供了有利工具。     

基于人工鱼群算法的高级综合生产计划优化研究

充分考虑了多产品多阶段多分销商企业生产过程各种影响因素及其相互关系后,构造了它的计划优化模型.在此基础上,应用人工鱼群算法,对该模型进行研究,并在计算机上得以仿真实现.仿真结果表明,该算法不受问题规模的限制,对于解决大规模复杂的问题更显示出其优越性,为企业生产计划优化提供了有利工具.     

一类含同工件流水线调度问题的优化研究

流水线调度问题是具有很强工程背景的典型NP完全问题,当其含有同工件时,批量和排序的相关性使得问题的求解更为复杂。文章有机结合模拟退火的概率突跳性和遗传算法的并行搜索结构,提出了处理一类含同工件流水线调度问题的混合优化策略。算法不仅能够动态缩小搜索空间以提高搜索效率,而且在保优策略的基础上利用重升温技术来增强克服陷入局部极小的能力,其有效性和快速性通过仿真得到了验证。     

错误流模型:硬件故障的软件传播建模与分析

无论是可靠性工程还是软件可靠性中的可靠性模型,都难以描述硬件故障在程序中的传播问题.首先建立了计算数据流模型,并以无穷存储机器的指令集为例,说明可以为任意程序建立计算数据流图.在计算数据流模型的基础上,进一步建立了错误流模型.把计算过程中的错误分成物理错误和传播错误两种,通过分析这两种错误的本质和传播规律,给出了6条有关错误传播的规则和2条独立定律.根据这些规则和定律,能够计算出在程序运行过程中,任意时刻在任意位置上出现错误的概率.最后以一个简单的无穷存储机器程序为例,简要地展示了错误流模型描述硬件故障在     

基于网格实现的汽轮机基础优化设计

工程优化设计往往需要进行大规模的数值计算,拥有大量闲置资源的网格环境为建立这种高性能计算平台提供了可能.但是网格资源的动态性、异构性和分布性的本质特征,阻碍了网格技术在工程应用上的普及.为了利用网格环境中大量的闲置资源来协同解决实际工程中复杂的优化设计问题,建立了一个4层结构的高性能网格计算平台,并利用Kriging近似模型,在该平台上开发了以减轻基础重量和降低基础振幅为目的的多目标汽轮机优化设计的网格算法.使用该算法,在网格平台上对两个汽轮机基础进行了优化设计,与序列线性规划方法的结果比较表明所开发的优化算法有较高的计算精度.还分析了当使用不同数量的计算节点时网格的加速情况,说明所发展的优化方法能够在网格环境中高效地运行,搭建的网格平台也适合于工程优化设计.     

Towards Efficient Execution of MPI Applications on the Grid: Porting and Optimization Issues

The message passing interface (MPI) is a standard used by many parallel scientific applications. It offers the advantage of a smoother migration path for porting applications from high performance computing systems to the Grid. In this paper Grid-enabled tools and libraries for developing MPI applications are presented. The first is MARMOT, a tool that checks the adherence of an application to the MPI standard. The second is PACX-MPI, an implementation of the MPI standard optimized for Grid environments. Besides the efficient development of the program, an optimal execution is of paramount importance for most scientific applications. We therefore discuss not only performance on the level of the MPI library, but also several application specific optimizations, e.g., for a sparse, parallel equation solver and an RNA folding code, like latency hiding, prefetching, caching and topology-aware algorithms.     

基于信誉度的网格资源质量优化

网格由分布在地理上的各种各样的计算机资源组成,用户在调用资源时,只会根据资源提供商所制定的条件来选择适合自己的资源,但是资源质量的优劣只有在至少用完一次后才可以得出结论.现今许多基于经济模型的资源调度策略只是在提高资源提供商的效益及资源调用时的负载平衡方面加以研究,而网格资源的质量并无过多深入的分析.文中提出的基于信誉度的网格资源调度算法,在用户与资源之间插入一个信誉度计算器,用户每调用一次便会对此资源进行一次信誉度评估.实验证明此算法不仅提高了资源提供商获得的效益,并且使得网格资源的质量得到进一步的优化.