基于滑动窗口的进化数据流聚类

提出了纳伪(false positive)和拒真(false negative)两种聚类特征指数直方图分别来支持纳伪误差和拒真误差窗口的聚类分析;然后,提出一种基于滑动窗口的数据流聚类方法.该方法在占用窗口大小的次线性内存空间前提下,及时保存最近数据记录的分布状况,从而实现对滑动窗口内的数据进行聚类.此外,它还可被扩展用于N-n窗口(滑动窗口的扩展模型)的数据聚类.实验采用KDD-CUP'99和KDD-CUP'98真实数据集以及变换高斯分布的人工数据集构造进化数据流.理论分析和     

Outlier Detection over Sliding Windows for Probabilistic Data Streams

Outlier detection is a very useful technique in many applications, where data is generally uncertain and could be described using probability. While having been studied intensively in the field of deterministic data, outlier detection is still novel in the emerging uncertain data field. In this paper, we study the semantic of outlier detection on probabilistic data stream and present a new definition of distance-based outlier over sliding window. We then show the problem of detecting an outlier over a set o...     

Improved Approximate Detection of Duplicates for Data Streams Over Sliding Windows

Detecting duplicates in data streams is an important problem that has a wide range of applications. In general,precisely detecting duplicates in an unbounded data stream is not feasible in most streaming scenarios,and,on the other hand,the elements in data streams are always time sensitive. These make it particular significant approximately detecting duplicates among newly arrived elements of a data stream within a fixed time frame. In this paper,we present a novel data structure,Decaying Bloom Filter(DBF),as an extension of the Counting Bloom Filter,that effectively removes stale elements as new elements continuously arrive over sliding windows. On the DBF basis we present an efficient algorithm to approximately detect duplicates over sliding windows. Our algorithm may produce false positive errors,but not false negative errors as in many previous results. We analyze the time complexity and detection accuracy,and give a tight upper bound of false positive rate. For a given space G bits and sliding window size W,our algorithm has an amortized time complexity of O((G/W))~(1/2). Both analytical and experimental results on synthetic data demonstrate that our algorithm is superior in both execution time and detection accuracy to the previous results.     

挖掘滑动窗口中的数据流频繁模式

随着数据流应用的不断增多,数据流环境下的数据挖掘技术受到了越来越多的关注.文章结合数据流的特点,提出一种新的基于滑动窗口的频繁模式挖掘算法:DSFPM.算法分块挖掘数据流,在内存中维持一个用于保存所有潜在的频繁模式信息的存储结构DSFPM-Tree,并在各个基本窗口进入滑动窗口后动态更新该存储结构.算法仅处理和保存各个基本窗口的临界频繁闭合项集,极大地提高了时间和空间效率.实验结果表明,该算法具有良好的性能.     

滑动窗口规模的动态调整算法

讨论当数据流系统的数据流流速或连续查询发生变化时,滑动窗口规模的动态调整问题.根据可用内存空间大小和连续查询需求,提出了3类动态调整滑动窗口规模的算法,实现了对连续查询3种服务质量级别的支持,提高了连续查询处理的效率和效果.理论分析与实验结果表明,提出的算法可以有效地应用于数据流系统.     

数据流上的复合滑动窗口聚集算法

基于滑动窗口的聚集查询是数据流研究领域的一个热点问题。在已有的研究工作中,聚集算法都是针对立即执行的连续查询提出的,这些算法均是当数据流新到一个元组立即计算一次聚集结果。而在实际应用中,连续查询有时采取的是周期执行方式。论文针对周期执行的连续查询提出了复合滑动窗口聚集算法,即数据流新到一个元组,将它插入到基本窗口中,当基本窗口被插满时计算一次聚集结果。给出了非增量式和增量式两种算法。理论分析和实验结果表明增量式算法具有较好的性能。     

分布式数据流连接查询算法

分布式处理是数据流管理系统发展的必然趋势。文章研究了分布式数据流的连接查询，提出DM3Join算法，它由2部分组成：一是通过分解并发的连接请求，合并相同的连接谓词，形成分布式查询操作算子；二是数据流在各分布式代理(Agent)中流转实现部分连接，并在查询引擎处组合成最终结果。DM3Join算法采用了一种类似路由表的结构执行窗口连接，由于可以共享中间结果，算法只需扫描数据1遍。分析和实验证明，该连接算法是高效的。     

基于数据概要描述的分布式数据流聚类模型与算法

数据流挖掘可有效解决大容量流式数据的知识发现问题,并已得到广泛研究.数据流的一个典型的例子是传感器采集的流式数据.然而,随着传感器网络的应用普及,这些流式数据在很多情况下是分布式采集和管理的,这就必然导致分布式地挖掘数据流的需求.分布式数据流挖掘的最大障碍是由分布式而导致的挖掘质量或者效率问题.为适应分布式数据流的聚类挖掘,探讨了分布式数据流的挖掘模型,并且基于该模型设计了对应的概要数据结构和关键的挖掘算法,给出了算法的理论评估或者实验验证.实验说明,提出的模型和算法可以有效地减少数据通信代价,并且能保证较高的全局模式的聚类质量.     

Continuous Monitoring of Distributed Data Streams over a Time-Based Sliding Window

In this paper we extend the study of algorithms for monitoring distributed data streams from whole data streams to a time-based sliding window. The concern is how to minimize the communication between individual streams and the root, while allowing the root, at any time, to report the global statistics of all streams within a given error bound. This paper presents communication-efficient algorithms for three classical statistics, namely, basic counting, frequent items and quantiles. The worst-case communication cost over a window is $O(\frac{k}{\varepsilon} \log\frac{\varepsilon N}{k})$ bits for basic counting, $O(\frac{k}{\varepsilon} \log\frac{N}{k})$ words for frequent items and $O(\frac{k}{\varepsilon^{2}} \log\frac{N}{k})$ words for quantiles, where k is the number of distributed data streams, N is the total number of items in the streams that arrive or expire in the window, and ε<1 is the given error bound. The performance of our algorithms matches and nearly matches the corresponding lower bounds. We also show how to generalize these results to streams with out-of-order data.     

基于操作符优先级的两种分布式数据流负载分配算法研究

在分布式数据流中的查询大多表现为连续查询形式,这种查询方式一旦被注册到流系统中后就一直存在,除非特意将其删除.由于流系统中的输入数据是源源不断到来的,因此数据流中的连续查询并不存在传统分布式数据库中查询任务的完成时间概念,反之,它则更关心查询结果的时间延迟.基于此,提出了两种最小化连续查询结果时间延迟的操作符负载分配策略,即PTDM算法和PPLB算法.实验结果表明,相比于其他一些操作符负载分配策略而言,这两种负载分配策略可以有效减小连续查询结果的时间延迟,从而提高分布式数据流的连续查询效率.     

分布式复式数据流的处理

在分布式数据流环境中,系统的通信带宽是一种瓶颈资源．在保证查询精度的前提下,为了有效地减少网络中数据流的传输量,提出了一种新的数据流传输方式,称为复式数据流．复式数据流方法是将分布式数据流系统中的原始数据流分组合并成复式数据流之后再进行传输．在定义了复式数据流的基础上,给出了复式数据流的生成算法,并且分析了基于复式数据流的查询操作的误差度,讨论了构造复式数据流的相关问题,最后通过实验验证了这种方法的有效性．     

面向大数据流的分布式索引构建

大数据流的高效存储与索引是当今数据领域的一大难点.面向带有时间属性的数据流,根据其时间属性,将数据流划分为连续的时间窗口,提出了基于双层B+树的分布式索引结构WB-Index.下层B+树索引基于窗口内流数据构建,索引构建过程结合基于排序的批量构建技术,进一步对时间窗口分片,将数据流接收、分片数据排序以及B+树构建并行化,提高了构建性能.上层B+树索引基于各时间窗口构建,结合时间窗口时间戳的递增性和无限性,提出了避免节点分裂的构建方法,减少了B+树分裂移动开销,提高了空间利用率和更新效率.WB-Index架构中,将流数据和索引分离,同时利用内存缓存尽可能多的双层B+索引和热点数据来提高查询性能.理论和实验结果表明,该分布式索引架构能够支持高效的实时数据流写入以及流数据查询,能够很好地应用于具有时间属性的数据流场景.     

基于分形技术的数据流突变检测算法

数据流上的突变检测技术由于其在风险分析、网络监测、趋势分析等领域广阔的应用前景而受到学术界和工业界越来越多的关注.为了在数据流上检测多个滑动窗口上的单调聚集函数值和非单调聚集函数值的突变,提出了基于分形技术的构建单调搜索空间的突变检测算法.首先给出了数据流上的分段分形模型,进而基于该模型设计了突变检测算法.该算法能够将突变检测处理时间复杂度从O(m)降为O(logm)(m为需要被检测的滑动窗口数目).提出的两种新颖的分段分形模型能够准确     

Optimization Algorithms for Distributed Queries

The efficiency of processing strategies for queries in a distributed database is critical for system performance. Methods are studied to minimize the response time and the total time for distributed queries. A new algorithm (Algorithm GENERAL) is presented to derive processing strategies for arbitrarily complex queries. Three versions of the algorithm are given: one for minimizing response time and two for minimizing total time. The algorithm is shown to provide optimal solutions under certain conditions.     

基于距离的分布式RFID数据流孤立点检测

RFID技术已广泛应用于实时监控、对象标识及跟踪等领域,及时发现被监控标签对象的异常状态显得十分重要.然而,由于无线通信技术的不可靠性及环境因素影响,RFID阅读器收集到的数据常常包含噪声.针对分布式RFID数据流的海量、易变、不可靠及分布等特点,提出了基于距离的局部流孤立点检测算法LSOD和基于近似估计的全局流孤立点检测算法GSOD.LSOD需要维护数据流结构CSL来识别安全内点,然后运用安全内点的特性来节省流数据的存储空间和查询时间.根据基于距离的孤立点定义,在中心节点上的全局孤立点是位于每个分布节点上孤立点集合的子集.GSOD采用抽样方法进行全局孤立点近似估计,以减少中心节点的通信量及计算负荷.实验表明,所给出的算法具有运行时间短、占用内存小、准确率高等特点.     

数据流上的预测聚集查询处理算法

实时数据流未来趋势的预测具有重要的实际应用意义.例如,在环境监测传感器网络中,通过对感知数据流进行预测聚集查询,观察者可以预测网络覆盖的区域在未来一段时间内的平均温度和湿度,以确定是否会发生异常事件.目前的研究工作多数集中在数据流上当前数据的查询,数据流上预测查询的研究工作还很少.采用多元线性回归方法,给出了数据流上的聚集值预测模型,提出了一种数据流预测聚集查询处理方法.当预测失败的次数大于预先给定的阈值时,给出了一种预测模型自动调整策略,以降低预测误差.还提出了滑动窗口的更新周期、数据流的流速对预测精度影响的数学模型.理论分析与实验结果表明,提出的预测聚集查询处理算法具有较高的性能,并且能够返回满足用户精度要求的预测查询结果.在实验中,采用TPC-H国际标准测试数据和TAO(tropical atmosphere ocean)测量的海洋表面空气温度数据来构造数据流.     

Media Streams Scheduling for Synchronization in Distributed Multimedia Systems

In this paper, we consider the problem of multimedia synchronization based on scheduling the transmission of multimedia documents in a networked environment. Assuming channels with different bandwidth and delay characteristics are established between the multimedia server and the client, we formulate the scheduling problem to ensure interstream and intrastream synchronization as a parallel processor scheduling problem. Since the heterogeneous parallel processor scheduling problem is NP-hard, we propose two heuristic algorithms with time complexity ofO(n log n+nm), wherenis the number of data units to be scheduled andmthe number of channels available. We also develop an enumerative algorithm to obtain the exact solutions. Extensive computational simulations reveal that the heuristics consistently obtain near-optimal solutions. From the simulation results, we also identify special structures of multimedia documents along with characteristics of the available channels which affect the relative performance of the algorithms.     

Improved Distributed Algorithms for SCC Decomposition

We study and improve the OBF technique [Barnat, J. and P.Moravec, Parallel algorithms for finding SCCs in implicitly given graphs, in: Proceedings of the 5th International Workshop on Parallel and Distributed Methods in Verification (PDMC 2006), LNCS (2007)], which was used in distributed algorithms for the decomposition of a partitioned graph into its strongly connected components. In particular, we introduce a recursive variant of OBF and experimentally evaluate several different implementations of it that vary in the degree of parallelism. For the evaluation we used synthetic graphs with a few large components and graphs with many small components. We also experimented with graphs that arise as state spaces in real model checking applications. The experimental results are compared with that of other successful SCC decomposition techniques [Orzan, S., “On Distributed Verification and Verified Distribution,” Ph.D. thesis, Free University of Amsterdam (2004); Fleischer, L.K., B. Hendrickson and A. Pinar, On identifying strongly connected components in parallel, in: Parallel and Distributed Processing, IPDPS Workshops, Lecture Notes in Computer Science 1800, 2000, pp. 505–511].     

Fast Algorithm for Walsh Hadamard Transform on Sliding Windows

This paper proposes a fast algorithm for Walsh Hadamard Transform on sliding windows which can be used to implement pattern matching most efficiently. The computational requirement of the proposed algorithm is about 1.5 additions per projection vector per sample, which is the lowest among existing fast algorithms for Walsh Hadamard Transform on sliding windows.