Design patterns for real-time distributed control system benchmarking

In this paper, we describe the design and development of a simulation–agent interface for real-time distributed control system benchmarking. This work is motivated by the need to test the feasibility of extending agent-based systems to the physical device level in manufacturing and other industrial automation systems. Our work focuses on the development of hybrid physical/simulation environment that can be used to perform tests at both the physical device level, as well as the planning and scheduling level of control. As part of this work, we have extended the proxy design pattern for this application. This paper focuses on the resulting software design pattern for distributed control system benchmarking and provides examples of its use in our hybrid physical/simulation environment.     

分布式文件系统在线性能测试平台研究与实现

目前由于缺乏针对分布式文件系统的性能测试工具,在很多研究中依然照搬传统的文件系统性能测试工具和方法,很难满足分布式文件系统性能测试的需求(如避免缓存的影响、多客户端并发测试等)。因此,我们研发了一个分布式文件系统在线性能测试平台,此平台通过提供多种大数据量测试负载,有效地避免了系统缓存和系统老化对测试结果的影响,同时支持在线对分布式文件系统执行性能测试,支持多客户端并发测试,并能够对测试结果进行可视化展示和分析。此外,平台也提供了很好的扩展性,满足多种测试需求。     

GridBench: A tool for the interactive performance exploration of Grid infrastructures

As Grids rapidly expand in size and complexity, the task of benchmarking and testing, interactive or unattended, quickly becomes unmanageable. In this article we describe the difficulties of testing/benchmarking resources in large Grid infrastructures and we present the software architecture implementation of GridBench, an extensible tool for testing, benchmarking and ranking of Grid resources. We give an overview of GridBench services and tools, which support the easy definition, invocation and management of tests and benchmarking experiments. We also show how the tool can be used in the analysis of benchmarking results and how the measurements can be used to complement the information provided by Grid Information Services and used as a basis for resource selection and user-driven resource ranking. In order to illustrate the usage of the tool, we describe scenarios for using the GridBench framework to perform test/benchmark experiments and analyze the results.     

Simulation-based benchmarking of production control schemes for complex manufacturing systems

In this paper, benchmarking efforts for production control approaches applied to complex manufacturing systems are described. Complex manufacturing systems are characterized by a large number of products, an over time changing product mix, sequence-dependent set-up times, unrelated parallel machines, a mix of different process types, and internal and external disturbances. In order to ensure comparison possibilities among different production control approaches, the usage of a simulation test-bed and a software architecture that allows for plug in of different production control software are suggested. Different application areas, the advantages, and also the limitations of the suggested approach are discussed.     

软件健壮性的包裹测试

软件的健壮性正变得越来越重要,目前还缺乏准确的、可重复利用的方法来衡量系统的健壮性。健壮性基准程序就是为了解决这一问题而提出的,它由一系列健壮性测试案例组成。本文以Linux为例,简要介绍了当今操作系统健壮性基准程序测试的主要方法,对用于分析测试结果的维度模型进行了分析,提出了软件健壮性的包裹测试方法,并用实例说明了其有效性。     

Smart CloudBench—A framework for evaluating cloud infrastructure performance

Cloud migration allows organizations to benefit from reduced operational costs, improved flexibility, and greater scalability, and enables them to focus on core business goals. However, it also has the flip side of reduced visibility. Enterprises considering migration of their IT systems to the cloud only have a black box view of the offered infrastructure. While information about server pricing and specification is publicly available, there is limited information about cloud infrastructure performance. Comparison of alternative cloud infrastructure offerings based only on price and server specification is difficult because cloud vendors use heterogeneous hardware resources, offer different server configurations, apply different pricing models and use different virtualization techniques to provision them. Benchmarking the performance of software systems deployed on the top of the black box cloud infrastructure offers one way to evaluate the performance of available cloud server alternatives. However, this process can be complex, time-consuming and expensive, and cloud consumers can greatly benefit from tools that can automate it. Smart CloudBench is a generic framework and system that offers automated, on-demand, real-time and customized benchmarking of software systems deployed on cloud infrastructure. It provides greater visibility and insight into the run-time behavior of cloud infrastructure, helping consumers to compare and contrast available offerings during the initial cloud selection phase, and monitor performance for service quality assurance during the subsequent cloud consumption phase. In this paper, we first discuss the rationale behind our approach for benchmarking the black box cloud infrastructure. Then, we propose a generic architecture for benchmarking representative applications on the heterogeneous cloud infrastructure and describe the Smart CloudBench benchmarking workflow. We also present simple use case scenarios that highlight the need for tools such as Smart CloudBench.     

BerlinMOD: a benchmark for moving object databases

This document presents a method to design scalable and representative moving object data (MOD) and two sets of queries for benchmarking spatio-temporal DBMS. Instead of programming a dedicated generator software, we use the existing Secondo DBMS to create benchmark data. The benchmark is based on a simulation scenario, where the positions of a sample of vehicles are observed for an arbitrary period of time within the street network of Berlin. We demonstrate the data generator’s extensibility by showing how to achieve more natural movement generation patterns, and how to disturb the vehicles’ positions to create noisy data. As an application and for reference, we also present first benchmarking results for the Secondo DBMS. Whereas the benchmark focuses on range queries, we demonstrate its ability to incorporate new future classes of queries by presenting a preliminary extension handling various nearest neighbour queries. Such a benchmark is useful in several ways: It provides well-defined data sets and queries for experimental evaluations; it simplifies experimental repeatability; it emphasizes the development of complete systems; it points out weaknesses in existing systems motivating further research. Moreover, the BerlinMOD benchmark allows one to compare different representations of the same moving objects.     

On Benchmarking Constraint Logic Programming Platforms. Response to Fernandez and Hill's “A Comparative Study of Eight Constraint Programming Languages over the Boolean and Finite Domains”

The comparative study published in this journal by Fernandez and Hill benchmarked some constraint programming systems on a set of well-known puzzles. The current article examines the positive and negative aspects of this kind of benchmarking.The article analyses some pitfalls in benchmarking, recalling previous published results from benchmarking different kinds of software, and explores some issues in comparative benchmarking of CLP systems.A benchmarking exercise should cover a broad set of representative problems and a broad set of programming constructs. This can be achieved using two kinds of benchmarking: Applications Benchmarking and Unit Testing. The article reports the authors' experiences with these two kinds of benchmarking in the context of the CHIC2 Esprit project. The benchmarks were used to unit test different features of the CLP system ECLiPSe and to compare application development with different high-level constraint platforms.The conclusion is that, in deciding which system to use on a new application, it is less useful to compare standard features of CLP systems, than to compare their relevant functionalities.     

A novel framework for software defined based secure storage systems

The Software Defined Systems (SDSys) paradigm has been introduced recently as a solution to reduce the overhead in the control and management operations of complex computing systems and to maintain a high level of security and protection. The main concept behind this technology is around isolating the data plane from the control plane. Building a Software Defined System in a real life environment is considered an expensive solution and may have a lot of risks. Thus, there is a need to simulate such systems before the real-life implementation and deployment. In this paper we present a novel experimental framework as a virtualized testbed environment for software defined based secure storage systems. Its also covers some related issues for large scale data storage and sharing such as deduplication. This work builds on the Mininet simulator, where its core components, the host, switch and the controller, are customized to build the proposed experimental simulation framework. The developed emulator, will not only support the development and testing of SD-based secure storage solutions, it will also serve as an experimentation tool for researchers and for benchmarking purposes. The developed simulator/emulator could also be used as an educational tool to train students and novice researchers.     

Unlabelled extra data do not always mean extra performance for semi-supervised fault prediction

Abstract: This research focused on investigating and benchmarking several high performance classifiers called J48, random forests, naive Bayes, KStar and artificial immune recognition systems for software fault prediction with limited fault data. We also studied a recent semi-supervised classification algorithm called YATSI (Yet Another Two Stage Idea) and each classifier has been used in the first stage of YATSI. YATSI is a meta algorithm which allows different classifiers to be applied in the first stage. Furthermore, we proposed a semi-supervised classification algorithm which applies the artificial immune systems paradigm. Experimental results showed that YATSI does not always improve the performance of naive Bayes when unlabelled data are used together with labelled data. According to experiments we performed, the naive Bayes algorithm is the best choice to build a semi-supervised fault prediction model for small data sets and YATSI may improve the performance of naive Bayes for large data sets. In addition, the YATSI algorithm improved the performance of all the classifiers except naive Bayes on all the data sets.     

数字水印的应用、性质及性能评测

The last ten years have seen rapidly growing in the field of digital watermarks. Meanwhile ,a large number of attacks appears as fast as new algorithms are proposed,which emphasizes the limits and weaknesses of those algo-rithms. The tools ,methodologies and benchmarking of assessing the current algorithms are almost non-existent. This paper reviews a number of applications of digital watermarking and their properties ,describes and classifies a numberof attacks on digital watermarks,and analyzes the Stirmark,Checkmark,Optimark benchmarking systems. At end,we give some comment on the watermarkinu research.     

Semantic BMS: Allowing usage of building automation data in facility benchmarking

Facility benchmarking and evaluation of facility performance are the crucial tasks in reaching efficient, economical and sustainable facility operation. Modern buildings are equipped with building automation systems (BAS) that contain vast numbers of various sensors that can be utilised in performance assessment. However, such systems lack convenient tools for data inspection, which limits their use in building performance and efficiency analysis and benchmarking especially on large sites. The paper presents a middleware layer designed to enrich BAS data with additional semantic information. As a semantic model, an adaptation of the Semantic Sensor Network (SSN) ontology for the field of building operation analysis is used. The middleware provides convenient interfaces for querying the model. The proposed system provides the facility managers with a convenient way to use the BAS data for benchmarking and decision support.     

Simulating heterogeneous behaviours in complex systems on GPUs

Agent Based Modelling (ABM) is an approach for modelling dynamic systems and studying complex and emergent behaviour. ABMs have been widely applied in diverse disciplines including biology, economics, and social sciences. The scalability of ABM simulations is typically limited due to the computationally expensive nature of simulating a large number of individuals. As such, large scale ABM simulations are excellent candidates to apply parallel computing approaches such as Graphics Processing Units (GPUs). In this paper, we present an extension to the FLAME GPU¹ [1] framework which addresses the divergence problem, i.e. the challenge of executing the behaviour of non-homogeneous individuals on vectorised GPU processors. We do this by describing a modelling methodology which exposes inherent parallelism within the model which is exploited by novel additions to the software permitting higher levels of concurrent simulation execution. Moreover, we demonstrate how this extension can be applied to realistic cellular level tissue model by benchmarking the model to demonstrate a measured speedup of over 4x.     

网络性能基准测试序列的边值生成法研究

边值分析是网络性能基准测试序列生成的一种重要方法,本文针对网络基准测试系统的特点,提出网络基准测试模型,并对模型中的实体与关系进行了抽象和描述;在此基础上分析模型中实体和关系属性的边值特性,并根据应用实例提出测试序列的边值分析生成算法。     

A Requirements Driven Framework for Benchmarking Semantic Web Knowledge Base Systems

A key challenge for the semantic Web is to acquire the capability to effectively query large knowledge bases. As there will be several competing systems, we need benchmarks that will objectively evaluate these systems. Development of effective benchmarks in an emerging domain is a challenging endeavor. In this paper, we propose a requirements driven framework for developing benchmarks for semantic Web knowledge base systems (SW KBSs). In this paper, we make two major contributions. First, we provide a list of requirements for SW KBS benchmarks. This can serve as an unbiased guide to both the benchmark developers and personnel responsible for systems acquisition and benchmarking. Second, we provide an organized collection of techniques and tools needed to develop such benchmarks. In particular, the collection contains a detailed guide for generating benchmark workload, defining performance metrics, and interpreting experimental results     

ECG databases for biometric systems: A systematic review

Computer-based biometric systems (CBBSs) individual recognition are expert and intelligent systems that are gaining increasing interest in many areas, such as securing financial systems, telecommunications and healthcare applications. The electrocardiogram (ECG) has been used as biometric feature for its low circumvention, large acceptability and uniqueness, thus being at the basis of several CBBSs. As ECG databases collected for clinical applications are not adequate for biometric applications, we have assisted to the development of other repositories of ECG, each one different from the others and highlighting certain issues of ECG-based biometric recognition. Through a systematic framework presented here, we quantitative analyse, evaluate and compare the acquisition hardware and the acquisition protocols of ECG databases available in literature and suited to develop CBBSs. Although the most recent ones, namely CYBHI and UofTDB, result the best for the acquisition hardware and the acquisition protocols, respectively, our survey shows that none is exhaustive for developing a robust and general enough CBBSs. The analysis also highlights the current lack of standardization in this field and the difficulty of performing an effective benchmarking activity. Since a publicly available database is essential for the research community in ECG-based CBBS to correctly assess the performance of existing algorithms or even commercial expert systems, we also discuss here the main features that an “optimal” repository for the intelligent application at hand.     

Evaluating ARM HPC clusters for scientific workloads

The power consumption of modern high‐performance computing (HPC) systems that are built using power hungry commodity servers is one of the major hurdles for achieving Exascale computation. Several efforts have been made by the HPC community to encourage the use of low‐powered system‐on‐chip (SoC) embedded processors in large‐scale HPC systems. These initiatives have successfully demonstrated the use of ARM SoCs in HPC systems, but there is still a need to analyze the viability of these systems for HPC platforms before a case can be made for Exascale computation. The major shortcomings of current ARM‐HPC evaluations include a lack of detailed insights about performance levels on distributed multicore systems and performance levels for benchmarking in large‐scale applications running on HPC. In this paper, we present a comprehensive evaluation of results that covers major aspects of server and HPC benchmarking for ARM‐based SoCs. For the experiments, we built an unconventional cluster of ARM Cortex‐A9s that is referred to as Weiser and ran single‐node benchmarks (STREAM, Sysbench, and PARSEC) and multi‐node scientific benchmarks (High‐performance Linpack (HPL), NASA Advanced Supercomputing (NAS) Parallel Benchmark, and Gadget‐2) in order to provide a baseline for performance limitations of the system. Based on the experimental results, we claim that the performance of ARM SoCs depends heavily on the memory bandwidth, network latency, application class, workload type, and support for compiler optimizations. During server‐based benchmarking, we observed that when performing memory intensive benchmarks for database transactions, x86 performed 12% better for multithreaded query processing. However, ARM performed four times better for performance to power ratios for a single core and 2.6 times better on four cores. We noticed that emulated double precision floating point in Java resulted in three to four times slower performance as compared with the performance in C for CPU‐bound benchmarks. Even though Intel x86 performed slightly better in computation‐oriented applications, ARM showed better scalability in I/O bound applications for shared memory benchmarks. We incorporated the support for ARM in the MPJ‐Express runtime and performed comparative analysis of two widely used message passing libraries. We obtained similar results for network bandwidth, large‐scale application scaling, floating‐point performance, and energy‐efficiency for clusters in message passing evaluations (NBP and Gadget 2 with MPJ‐Express and MPICH). Our findings can be used to evaluate the energy efficiency of ARM‐based clusters for server workloads and scientific workloads and to provide a guideline for building energy‐efficient HPC clusters. Copyright © 2015 John Wiley & Sons, Ltd.     

Graph database benchmarking on cloud environments with XGDBench

Online graph database service providers have started migrating their operations to public clouds due to the increasing demand for low-cost, ubiquitous graph data storage and analysis. However, there is little support available for benchmarking graph database systems in cloud environments. We describe XGDBench which is a graph database benchmarking platform for cloud computing systems. XGDBench has been designed with the aim of creating an extensible platform for graph database benchmarking which makes it suitable for benchmarking future HPC systems. We extend the Yahoo! Cloud Serving Benchmark (YCSB) to the area of graph database benchmarking by creation of XGDBench. The benchmarking platform is written in X10 which is a PGAS language intended for programming future HPC systems. We describe the architecture of the XGDBench and explain how it differs from the current state-of-the-art. We conduct performance evaluation of five famous graph data stores AllegroGraph, Fuseki, Neo4j, OrientDB, and Titan using XGDBench on Tsubame 2.0 HPC cloud environment.     

使用基准测试促进研究--对软件工程的挑战

计算机科学中,基准测试已经用来比较计算机系统性能、检索算法信息、数据库和其他一些技术。在研究领域,基准测试的创建和普遍使用经常伴随着快速的技术进步和团队建设。这些形成了科学学科的基准测试理论。基于这个理论,通过团队工作的方式定义基准测试来挑战软件工程研究,变得越来越具有科学性和需要。本文介绍一个成功使用基准测试促进研究状况的个案研究——逆向工程。