Querying fragmented relations in a distributed database

Summary The paper discusses the query optimization problem in a distributed database system supporting a complex fragmentation schema, in which different fragments can share the same set of data. The fragmentation and the distribution of data are assumed to be transparent to the user, who queries the database in terms of the global relations. The optimization of queries stated on a single fragmented relation is considered, and a method for its solution is presented. The method is based on an algorithm which calculates a set of virtual fragments that can alternatively be used to answer the query.     

Monitoring distributed real-time systems

This paper emphasizes the power ofmonitoring of distributed real-time systems as a promising tool for both scientific work and practical purposes. Starting out from a number of well-known problems with today's (industrial) real-time systems, a classification of remedial monitoring applications is given. The most important features of a monitoring system suitable for such purposes are discussed and related to the current research into monitoring of (general) distributed systems. Finally, some of the resulting conceptual issues underlying our prototype VTA monitoring system—currently being under development at our department—are presented.     

Maintaining sliding window skylines on data streams

The skyline of a multidimensional data set contains the "best" tuples according to any preference function that is monotonic on each dimension. Although skyline computation has received considerable attention in conventional databases, the existing algorithms are inapplicable to stream applications because 1) they assume static data that are stored in the disk (rather than continuously arriving/expiring), 2) they focus on "one-time" execution that returns a single skyline (in contrast to constantly tracking skyline changes), and 3) they aim at reducing the I/O overhead (as opposed to minimizing the CPU-cost and main-memory consumption). This paper studies skyline computation in stream environments, where query processing takes into account only a "sliding window" covering the most recent tuples. We propose algorithms that continuously monitor the incoming data and maintain the skyline incrementally. Our techniques utilize several interesting properties of stream skylines to improve space/time efficiency by expunging data from the system as early as possible (i.e., before their expiration). Furthermore, we analyze the asymptotical performance of the proposed solutions, and evaluate their efficiency with extensive experiments.     

Coordination for fragmented loops and scopes in a distributed business process

As partner relationships become more dynamic and global boundaries give way to a more agile and dynamic environment, the ability to distribute one's processes in an agile manner becomes increasingly important. Such processes may need to be split not only along their explicit dependencies but also along more complex behavior such as recovery behavior and loops. The resulting process fragments can be distributed and wired together, recreating the execution semantics of the original process model. In earlier work, we presented BPEL fragmentation covering data and explicit control dependencies. We now extend the approach to handle fragmenting loops and scopes. Maintaining the focus on standards and maximizing extensibility of Web service runtimes and standards, the solution defines and uses two new coordination protocols that plug into the WS-Coordination framework. The approach uses the standards as much as is feasible and addresses the remaining required functionality by providing architected extensions. This results in layered approach that maximizes transparency and interoperability. After defining the fragmentation approach for scopes and loops, an implementation is presented that extends the Active Endpoints BPEL engine and a WS-Coordination system. A detailed example is used to illustrate how the protocols are used at runtime to enable the coordinator and the process fragments to recreate the behavior of the original, unsplit process model.     

Continuous monitoring of skylines over uncertain data streams

Uncertain data are inevitable in many applications due to various factors such as the limitations of measuring equipment and delays in data updates. Although modeling and querying uncertain data have recently attracted considerable attention from the database community, there are still many critical issues to be resolved with respect to conducting advanced analysis on uncertain data. In this paper, we study the execution of the probabilistic skyline query over uncertain data streams. We propose a novel sliding window skyline model where an uncertain tuple may take the probability to be in the skyline at a certain timestamp t. Formally, a Wp-Skyline(p, t) contains all the tuples whose probabilities of becoming skylines are at least p at timestamp t. However, in the stream environment, computing a probabilistic skyline on a large number of uncertain tuples within the sliding window is a daunting task in practice. In order to efficiently calculate Wp-Skyline, we propose an efficient and effective approach, namely the candidate list approach, which maintains lists of candidates that might become skylines in future sliding windows. We also propose algorithms that continuously monitor the newly incoming and expired data to maintain the skyline candidate set incrementally. To further reduce the computation cost of deciding whether or not a candidate tuple belongs to the skyline, we propose an enhanced refinement strategy that is based on a multi-dimensional indexing structure combined with a grouping-and-conquer strategy. To validate the effectiveness of our proposed approach, we conduct extensive experiments on both real and synthetic data sets and make comparisons with basic techniques.     

Diverse nearest neighbors queries using linear skylines

k-nearest neighbor (k-NN) queries are well-known and widely used in a plethora of applications. However, in the original definition of k-NN queries there is no concern regarding diversity of the answer set with respect to the user’s interests. For instance, travelers may be looking for touristic sites that are close to where they are, but that would also lead them to see different parts of the city. Likewise, if one is looking for restaurants close by, it may be more interesting to learn about restaurants of different categories or ethnicities which are nonetheless relatively close. The interesting novel aspect of this type of query is that there are two competing criteria to be optimized: closeness and diversity. We propose two approaches that leverage the notion of linear skyline queries in order to find the k diverse nearest neighbors within a radius r from a given query point, or (k, r)-DNNs for short. Our proposed approaches return a relatively small set containing all optimal solutions for any linear combination of the weights a user could give to the two competing criteria, and we consider three different notions of diversity: spatial, categorical and angular. Our experiments, varying a number of parameters and exploring synthetic and real datasets, in both Euclidean space and road networks, respectively, show that our approaches are several orders of magnitude faster than a straightforward approach.     

Monitoring and debugging distributed realtime programs

In this paper we describe the design and implementation of an integrated monitoring and debugging system for a distributed real-time computer system. The monitor provides continuous, transparent monitoring capabilities throughout a real-time system's lifecycle with bounded, minimal, predictable interference by using software support. The monitor is flexible enough to observe both high-level events that are operating system- and application-specific, as well as low-level events such as shared variable references. We present a novel approach to monitoring shared variable references that provides transparent monitoring with low overhead. The monitor is designed to support tasks such as debugging realtime applications, aiding real-time task scheduling, and measuring system performance. Since debugging distributed real-time applications is particularly difficult, we describe how the monitor can be used to debug distributed and parallel applications by deterministic execution replay.     

Probabilistic skylines on uncertain data: model and bounding-pruning-refining methods

Uncertain data are inherent in some important applications. Although a considerable amount of research has been dedicated to modeling uncertain data and answering some types of queries on uncertain data, how to conduct advanced analysis on uncertain data remains an open problem at large. In this paper, we tackle the problem of skyline analysis on uncertain data. We propose a novel probabilistic skyline model where an uncertain object may take a probability to be in the skyline, and a p-skyline contains all objects whose skyline probabilities are at least p (0 < p ≤ 1). Computing probabilistic skylines on large uncertain data sets is challenging. We develop a bounding-pruning-refining framework and three algorithms systematically. The bottom-up algorithm computes the skyline probabilities of some selected instances of uncertain objects, and uses those instances to prune other instances and uncertain objects effectively. The top-down algorithm recursively partitions the instances of uncertain objects into subsets, and prunes subsets and objects aggressively. Combining the advantages of the bottom-up algorithm and the top-down algorithm, we develop a hybrid algorithm to further improve the performance. Our experimental results on both the real NBA player data set and the benchmark synthetic data sets show that probabilistic skylines are interesting and useful, and our algorithms are efficient on large data sets.     

Monitoring persistent items in the union of distributed streams

A persistent item in a stream is one that occurs regularly in the stream without necessarily contributing significantly to the volume of the stream. Persistent items are often associated with anomalies in network streams, such as botnet traffic and click fraud. While it is important to track persistent items in an online manner, it is challenging to zero-in on such items in a massive distributed stream. We present the first communication-efficient distributed algorithms for tracking persistent items in a data stream whose elements are partitioned across many different sites. We consider both infinite window and sliding window settings, and present algorithms that can track persistent items approximately with a probabilistic guarantee on the approximation error. Our algorithms have a provably low communication cost, and a low rate of false positives and false negatives, with a high probability. We present detailed results from an experimental evaluation that show the communication cost is small, and that the false positive and false negative rates are typically much lower than theoretical guarantees.     

Taking the Big Picture: representative skylines based on significance and diversity

The skyline is a popular operator to extract records from a database when a record scoring function is not available. However, the result of a skyline query can be very large. The problem addressed in this paper is the automatic selection of a small number \((k)\) of representative skyline records. Existing approaches have only focused on partial aspects of this problem. Some try to identify sets of diverse records giving an overall approximation of the skyline. These techniques, however, are sensitive to the scaling of attributes or to the insertion of non-skyline records into the database. Others exploit some knowledge of the record scoring function to identify the most significant record, but not sets of records representative of the whole skyline. In this paper, we introduce a novel approach taking both the significance of all the records and their diversity into account, adapting to available knowledge of the scoring function, but also working under complete ignorance. We show the intractability of the problem and present approximate algorithms. We experimentally show that our approach is efficient, scalable and that it improves existing works in terms of the significance and diversity of the results.     

High-level distributed coordination via distributed Haskell

With the boom of Networks and Internet, more and more applications have an inherent distributed character. Distributed functional language can support high-level distributed coordination with automatic management of many coordination aspects, due to its features like good abstraction mechanisms, a high-level computation language, and sophisticated type system. This paper briefly reviews the features and benefits of functional language Glasgow distributed Haskell （GdH）. It also compares GdH with object-oriented distributed languages. It concludes that GdH potentially offers significant benefits for constructing high-level distributed coordination.     

Materialization of fragmented views in multidimensional databases

The most effective technique to enhance performances of multidimensional databases consists in materializing redundant aggregates called views. In the classical approach to materialization, each view includes all and only the measures of the cube it aggregates. In this paper we investigate the benefits of materializing views in vertical fragments, aimed at minimizing the workload response time. We formalize the fragmentation problem as a 0–1 integer linear programming problem, which is then solved by means of a standard integer programming solver to determine the optimal fragmentation for a given workload. Finally, we demonstrate the usefulness of fragmentation by presenting a large set of experimental results based on the TPC-H benchmark.     

MIL primitives for querying a fragmented world

In query-intensive database application areas, like decision support and data mining, systems that use vertical fragmentation have a significant performance advantage. In order to support relational or object oriented applications on top of such a fragmented data model, a flexible yet powerful intermediate language is needed. This problem has been successfully tackled in Monet, a modern extensible database kernel developed by our group. We focus on the design choices made in the Monet interpreter language (MIL), its algebraic query language, and outline how its concept of tactical optimization enhances and simplifies the optimization of complex queries. Finally, we summarize the experience gained in Monet by creating a highly efficient implementation of MIL. Received November 10, 1998 / Accepted March 22, 1999     

A graph-based optimization algorithm for fragmented image reassembly

We propose a graph-based optimization framework for automatic 2D image fragment reassembly. First, we compute the potential matching between each pair of the image fragments based on their geometry and color. After that, a novel multi-piece matching algorithm is proposed to reassemble the overall image fragments. Finally, the reassembly result is refined by applying the graph optimization algorithm. We perform experiments to evaluate our algorithm on multiple torn real-world images, and demonstrate the robustness of this new assembly framework outperforms the existing algorithms in both reassembly accuracy (in handling accumulated pairwise matching error) and robustness (in handling small image fragments).     

A simple distributed garbage collector for distributed real-time Java

The use of real-time distribution middleware programmed with high-level languages like Java is becoming of increasing interest in next generation applications. Technology like Java’s Remote Method Invocation (RMI) paves the way towards these new distributed horizons. RMI offers many high-level abstractions useful for distributed application programmers to reduce their development times. One of these abstractions is a distributed garbage collector (DGC) that removes unreachable remote objects from the distributed ecosystem. However, in real-time Java, distributed garbage collection is underspecified and it introduces unbounded indeterminism on end-to-end real-time Java communications. This article analyzes this problem proposing a simple characterization for a predictable real-time distributed garbage collector (RT-DGC). The approach requires support from the middleware infrastructure that implements the abstraction but it also introduces bounded overhead. The article provides insight on the performance that RT-DGC offers to a distributed real-time Java application and the extra overheads due to the intrinsic cost of this abstraction.     

分布式数据库中查询处理的新方法研究

在分布式数据库系统中,由于数据的分布和冗余,使得分布式查询处理增加了许多新的内容和复杂性,通过分析现有分布式数据库查询处理技术,根据应用实际提出一种新的查询处理方法,该方法通过将常用查询结果存储在本地来减少查询时的数据传输量,从而缩短了响应时间.实验证明了该方法是有效的.     

Control of complex distributed systems with distributed intelligent agents

Control of spatially distributed systems is a challenging problem because of their complex nature, nonlinearity, and generally high order. The lack of accurate and computationally efficient model-based techniques for large, spatially distributed systems leads to challenges in controlling the system. Agent-based control structures provide a powerful tool to manage distributed systems by utilizing (organizing) local and global information obtained from the system. A hierarchical, agent-based system with local and global controller agents is developed to control networks of interconnected chemical reactors (CSTRs). The global controller agent dynamically updates local controller agent’s objectives as the reactor network conditions change. One challenge posed is control of the spatial distribution of autocatalytic species in a network of reactors hosting multiple species. The multi-agent control system is able to intelligently manipulate the network flow rates such that the desired spatial distribution of species is achieved. Furthermore, the robustness and flexibility of the agent-based control system is illustrated through examples of disturbance rejection and scalability with respect to the size of the network.     

Efficient distributed SAT and SAT-based distributed Bounded Model Checking

SAT-based Bounded Model Checking (BMC), though a robust and scalable verification approach, still is computationally intensive, requiring large memory and time. Even with the recent development of improved SAT solvers, the memory limitation of a single server rather than time can become a bottleneck for doing deeper BMC search for large designs. Distributing computing requirements of BMC over a network of workstations can overcome the memory limitation of a single server, albeit at increased communication cost. In this paper, we present (a) a method for distributed SAT over a network of workstations using a Master/Client model where each Client workstation has an exclusive partition of the SAT problem and uses knowledge of partition topology to communicate with other Clients, (b) a method for distributing SAT-based BMC using the distributed SAT. For the sake of scalability, at no point in the BMC computation does a single workstation have all the information. We experimented on a network of heterogeneous workstations interconnected with a standard Ethernet LAN. To illustrate, on an industrial design with ∼13 K FFs and ∼0.5 million gates, the non-distributed BMC on a single workstation (with 4 GB memory) ran out of memory after reaching a depth of 120; on the other hand, our SAT-based distributed BMC over 5 similar workstations was able to go up to 323 steps with a communication overhead of only 30%.     

Gain of optical distributed sensing in distributed parameter systems

A real time holographic sensing technique is introduced and its advantages arc investigated from the filtering and control point of view. The feature of holographic sensing is its capability to make distributed measurements of the position and velocity of moving objects, such as a vibrating flexible space structure. This study is based upon the distributed parameter models of linear time-invariant systems, particularly including the linear oscillator equations describing the vibration of large flexible space structures. The general conclusion is that application of optical distributed sensors brings gain in the situation where Kalman filtering is necessary for state estimation. In this case, both transient and steady stale filtering error covariance becomes smaller. This in turn results in smaller cost in the LQG problem.