Towards efficient data search and subsetting of large-scale atmospheric datasets

Discovering the correct dataset in an efficient fashion is critical for effective simulations in the atmospheric sciences. Unlike text-based web documents, many of the large scientific datasets often contain binary encoded data that is hard to discover using popular search engines. In the atmospheric sciences, there has been a significant growth in public data hosting services. However, the ability to index and search has been limited by the metadata provided by the data host. We have developed an infrastructure-Atmospheric Data Discovery System (ADDS)-that provides an efficient data discovery environment for observational datasets in the atmospheric sciences. To support complex querying capabilities, we automatically extract and index fine-grained metadata. Datasets are indexed based on periodic crawling of popular sites and also of files requested by the users. Users are allowed to access subsets of a large dataset through our data customization feature. Our focus is the overall architecture, data subsetting scheme, and a performance evaluation of our system.     

On the distributed orchestration of stochastic discrete event simulations

Discrete event simulations are a powerful technique for modeling stochastic systems with multiple components where interactions between these components are governed by the probability distribution functions associated with them. Complex discrete event simulations are often computationally intensive with long completion times. This paper describes our solution to the problem of orchestrating the execution of a stochastic, discrete event simulation where computational hot spots evolve spatially over time. Our performance benchmarks report on our ability to balance computational loads in these settings. Copyright © 2013 John Wiley & Sons, Ltd.     

Deploying the NaradaBrokering Substrate in Aiding Efficient Web and Grid Service Interactions

NaradaBrokering has been developed as the messaging infrastructure for collaboration, peer-to-peer, and Grid applications. The value of NaradaBrokering in the context of Grid and Web Services has been clear for some time. NaradaBrokering-combined with further extensions to, and testing of, its existing capabilities - can also take advantage of the maturing of Web Service standards and specifications to build very powerful general mechanisms to deploy and integrate it with general Web Services. This paper describes a framework to integrate the NaradaBrokering substrate with Web Services.     

Failure‐resilient real‐time processing of health streams

The ability to analyze streaming data in real time is vital in systems that process data from health sensors. These systems need to build and maintain state, as well as preserve this state during system failures. In this work, we introduce a fault‐tolerance scheme designed for the Granules stream processing system. We work with two distinct health stream datasets: thorax extension and electroencephalogram (EEG) signal analysis. We have developed a monitoring program to track trends in the thorax extension dataset and a classification system for the EEG dataset, which allows us to determine user intent from EEG signals. Using these two motivating applications, we have explored several approaches to fault tolerance through replication, developing a hybrid approach that is particularly suited to health streams. Copyright © 2014 John Wiley & Sons, Ltd.     

Enabling hierarchical dissemination of streams in content distribution networks

In streaming systems the content distribution network routes streams based on interests registered by the consuming entities. In hierarchical streaming, the dissemination is also predicated on the resolution of hierarchical dependencies between various streams. Entities specify explicit wildcards, in addition to the implicit ones in place, to further control the types of streams within a given hierarchy that should be routed to them. This paper presents an analysis and performance evaluation of three different algorithms for hierarchical streaming. In our evaluation of these algorithms we are especially interested in three factors: performance, ability to cope with flux, and memory consumption. Comprehensive benchmarks for these algorithms, in this paper, will enable system designers to harness the best algorithm that satisfies their hierarchical streaming requirements. Copyright © 2011 John Wiley & Sons, Ltd.     

Algorithms and the Grid

We review the impact of Grid Computing and Web Services on scientific computing, stressing the importance of the “data-deluge” that is driven by deployment of new instruments, sensors and satellites. This implies the need to integrate the naturally distributed data sources with large simulation engines offering parallel low latency communication and so to integrate parallel and Grid computing paradigms. We start with an overview of these and the evolving service architectures. We illustrate the identified areas of interest for Algorithms and the Grid with the specific example of SERVOGrid that supports earthquake science research. We comment on the appropriate messaging infrastructure for Grids and data assimilation and contrast it with MPI.     

Adaptive heterogeneous language support within a cloud runtime

Cloud runtimes are an effective method of distributing computations, but can force developers to use the runtime’s native language for all computations. We have extended the Granules cloud runtime with a bridge framework that allows computations to be written in C, C++, C#, Python, and R. We have additionally developed a diagnostics system which is capable of gathering information on system state, as well as modifying the underlying bridge framework in response to system load. Given the dynamic nature of Granules computations, which can be characterized as long-running with intermittent CPU bursts that allow a state to build up during successive rounds of execution, these bridges need to be bidirectional and the underlying communication mechanisms decoupled, robust and configurable. Granules bridges handle a number of different programming languages and support multiple methods of communication such as named pipes, unnamed pipes, and sockets. This choice of underlying communication mechanisms allows limited resources, such as sockets, to remain available for use by the runtime.