基于概率数据流的有效聚类算法

提出一种在概率数据流上进行聚类的有效方法P-Stream.P-Stream针对数据流上的概率元组提出强簇、过渡簇和弱簇的概念,设计一种有效的在线候选簇选择策略,为每个不断到达的数据元组合理地找到可能归属的簇,并在每个检查点存储微簇快照,以便离线进一步高层聚类和演化分析.最后设计一个“积极”的二层聚类模型来判断现有的第1层聚类模型是否还适应数据流中最近到达的概率元组.实验采用KDD-CUP’98和KDD-CUP’99真实数据集以及变换高斯分布的人工数据集构造概率数据流.实验结果表明,P-Stream具有良好的聚类质量、较快的处理速度,能够有效地适应数据演化情况.     

基于免疫原理的不确定数据流聚类算法

提出一种基于免疫原理、对不确定数据流进行聚类的算法——IUMicro。IUMicro针对不确定数据流上元组级不确定性问题,引入动态更新以适应数据变化的免疫模型,其中包括一种有效的在线收集数据流统计信息的B细胞特征结构及其更新策略。为兼顾元组存在概率与元组间的距离两方面因素,定义概率识别半径,为每个不断到达的数据元组找到合理的候选簇。离线聚类根据免疫细胞识别区域的空间关系,进行任意形状的无监督聚类。实验结果表明,IUMicro能有效抑制噪声,具有良好的聚类质量和较快的处理速度。     

一种不确定数据流子空间聚类算法

针对不确定数据流上的聚类问题提出一种不确定数据流子空间聚类算法UDSSC.该算法使用滑动窗口机制接收新到达的数据,剔除陈旧的数据;还引入子空间簇生成策略和新型离群点机制;系统建立了三个缓冲区分别存储新到来的元组、要进行聚类的元组和离群点元组,以此获得高质量的聚类结果.实验表明,UDSSC算法与同类型算法相比,具有更好的聚类效果、更低的时间复杂度和更强的扩展性.     

基于动态可调衰减滑动窗口的变速数据流聚类算法

在数据流聚类算法中,滑动窗口技术可以及时淘汰历史元组、只关注近期元组,从而改善数据流的聚类效果。如果同时数据流流速无规律地随时间动态变化,原来单纯的滑动窗口技术在解决这类问题时存在缺陷,所以,在充分考虑了滑动窗口大小和数据流流速之间关系的前提下,提出了基于动态可调衰减滑动窗口的变速数据流聚类算法。该算法对历史元组和近期元组分别赋予一定的权重进行处理,然后依据数据流流速的不同函数改变窗口的大小,从而实现数据流的聚类。提出了该数据流聚类算法的数据结构——变异数据流聚类的数据结构。通过真实数据和模拟数据来构造动态变速数据流从而作为验证算法的原始数据。实验结果表明,与Clu Stream聚类算法相比,该方法具有较高的聚类质量、较小的内存开销和较少的聚类处理时间。     

基于维度最大熵数据流聚类的异常检测方法

针对传统数据流聚类算法聚类信息损失大、不准确的缺点, 提出一种基于维度最大熵的数据流聚类算法. 采用动态数据直方图将数据维度划分为不同的维度组, 计算各维度最大熵划分维度空间簇, 将相同维度簇的数据聚集成微簇, 通过比较微簇的信息熵大小及其分布特点实现数据流的异常检测. 该方法提升了聚类速度, 克服了传统数据流聚类算法信息丢失的缺点. 实验结果表明, 所提出算法能够提高数据流异常检测的准确性和有效性.

     

基于混合差分进化的滑动窗口数据流聚类算法研究

针对传统的基于滑动窗口的数据流聚类算法存在的算法执行效率低、聚类质量较差等缺点,提出了一种基于混合差分进化的滑动窗口数据流聚类算法。该算法将数据流聚类过程分为两个部分：在线的时序窗口数据信息微簇特征向量生成和离线阶段的聚类优化。对在线生成的微簇进行微簇集合的更新与维护,利用改进的粒子群算法对离线的微簇数据信息进行适应度值的计算,将种群分为优势子种群和普通子种群,然后利用个体适应度值和平均适应度值的判别来生成当前个体环境的最优候选解,并迭代地对个体进行进化,输出具有最优适应度值的聚类集合,完成对数据流的聚类。仿真实验结果表明,算法在对数据流执行聚类时具有较高的执行效率,并且最后聚类的质量较好,算法实用性强。     

非一致性数据库的概率查询重写

结合概率数据库技术，以元组匹配所产生的聚类为基础，提出了一种新的基于聚类的非一致性数据的概率方法。基于可信聚类，给出了基本的查询重写技术，在有聚集的查询中，考虑了合适的元组概率、区间值、期望值。在不进行程序预处理的情况下，“重写”能被商业数据库系统有效地优化和执行，采用不一致性数据的区分度和数据库大小去理解其适应性，并使用了TPC-H基准的数据和查询。实验显示了该方法的有效性。     

一种存在级不确定数据流聚类算法

针对不确定数据流聚类算法——EMicro在聚类结果的精确性和聚类中数据的概率相似度方面的不足,提出一种新基于存在级的不确定数据流算法——UDs Stream。该算法通过设置概率阈值,将不确定性高的点和不确定性低的点分开处理,提高聚类结果的准确性和聚类中数据概率相似度。同时引入窗口和密度的方法,设置动态密度阈值,能够动态掌握数据流的分布特征,使聚类过程有更好的灵活性。实验结果证明,与EMicro相比,UDs Stream算法聚类效果更好。     

基于数据流的概念聚类

分析二部图的二元组和概念聚类问题之间的关系，在此基础上结合数据流的特点，提出一种适用于对象属性为布尔型的数据流概念聚类算法。将数据流分段，对每一批到来的数据流，生成局部的近似极大ε二元组集合，对全局的近似极大ε二元组集合进行更新，从而有效地对整个数据流进行聚类。实验结果表明，该算法具有良好的时间效率和空间效率。     

基于滑动窗口密度聚类的数据流偏倚采样算法

对于移动计算领域的移动对象轨迹数据流的管理,最普遍采用的技术手段是采样技术,而传统的均匀采样易丢失一些关键的变化数据,造成信息丢失现象.针对这一问题,提出一种基于概率密度聚类的数据流偏倚采样算法.该算法在滑动窗口模型下,充分利用了轨迹数据流自身的分布特性,结合偏倚采样算法思想克服了均匀采样的数据丢失问题.算法首先采用基于数据存在密度的聚类技术将滑动窗口划分为强簇、弱簇和过度簇,然后针对不同的簇给予不同的采样率,进行偏倚采样,进而得到最终的数据流摘要.经过实际数据集的实验检测,证明算法较好地保证了采样质量,并具有较快的数据处理能力.     

基于统计的无阻塞连接算法

数据流上的关系查询处理技术是数据库研究领域的一大热点。优化无阻塞连接算法的关键在于提高内存连接阶段的效率。当内存空间满时,需要将内存数据刷新到外存相应分区,良好的刷新策略对于改进算法的性能至关重要。利用数据分布的特征,对关系连接的输出流,使用基于统计的方法,查找使用频率最低的元组,将使用频率较低的元组刷新到外存,以提高内存数据的效率。基于统计分析策略提高了刷新策略的准确性和效率及算法的适用范围。     

Distributed stream join query processing with semijoins

This paper addresses the distributed stream processing of window-based multi-way join queries considering the semijoin as a key join operator. In distributed stream processing, data streams arriving at remote sites need to be shipped to the processing site for query execution. This typically introduces high communication overhead. Our observation is that semijoin, effective in reducing communication overhead in distributed database query processing, can be also effective in distributed stream query processing. The challenge, however, lies in the streaming nature of the tuples, as it requires continuous and incremental processing of an unbounded sequence of tuples instead of one-time processing of a set of stored tuples. This paper describes our comprehensive work done to address the challenge. Specifically, we first propose a distributed stream join processing model that handles the issue of network delays introduced from the shipment of data streams, and allows for efficient batch processing. Then, based on the model, we propose join algorithms in a multi-way join case: first, one-way join algorithms for different combinations of join placement and join method and, then, multi-way join algorithms assuming linear join ordering. Regarding the join method, two distributed join methods are introduced: (1) simple join, in which full tuples are forwarded to the query processing site and (2) semijoin-based join, in which partial tuples are forwarded. A semijoin-based join can be executed with different possible semijoin strategies which incur different communication overheads. We present a complete set of join algorithms considering all possible semijoin strategies, and propose an optimization algorithm. The join algorithms are executed continuously in an incremental manner as tuples arrive, and never ship tuples redundantly. The optimization algorithm constructs an efficient multi-way join plan by using a greedy heuristic which adds to the plan one stream with the minimum join execution cost in each step. Through extensive experiments, we conduct comparative studies of the performance among the proposed one-way join algorithms and the efficiency of the generated plan between the optimization algorithm based on the greedy heuristic and the exhaustive search, respectively.     

面向表数据发布隐私保护的贪心聚类匿名方法

为了防范隐私泄露,表数据一般需要匿名处理后发布.现有匿名方案较少分类考察准标识属性概化,并缺少同时考虑信息损失量和时间效率的最优化.利用贪心法和聚类划分的思想,提出一种贪心聚类匿名方法：分类概化准标识属性,并分别度量其信息损失,有利于减小并合理评价信息损失.对元组间距离和元组与等价类距离,建立与最小合并概化信息损失值正相关的距离定义,聚类过程始终选取具有最小距离值的元组添加,从而保证信息损失总量趋于最小.按照k值控制逐一聚类,实现等价类均衡划分,减少了距离计算总量,节省了运行时间.实验结果表明,该方法在减少信息损失和运行时间方面是有效的.     

面向不确定数据流的近似ER-Topk查询处理

随着移动互联网的快速发展以及信息技术的普遍应用,在许多应用中都产生了海量、不确定性数据,包括金融、军事、位置服务、医疗以及气象等。然而,传统的确定性数据管理方法很难管理不确定数据,亟需开发新型数据管理方法。可能世界模型被广泛用于为不确定数据建模,通过该模型可以衍生出诸多确定性的可能世界实例。不确定性数据流是指高速到达的海量不确定元组序列,因而不确定数据流管理比不确定性静态数据管理更具挑战性。面向于不确定数据流的ER-Topk查询是一个典型问题,但是处理复杂度高。提出一种近似算法来处理该查询,具有较小的空间复杂度;同时,还通过搜索策略优化来进一步提升查询处理效率。实验结果验证了所提方法的有效性和高效性。     

A dynamic assignment scheduling algorithm for big data stream processing in mobile Internet services

As the huge number of mobile devices (e.g., smart phones, tablets and netbooks) increases, more and more people choose to use the Internet services financed by mobile Internet service providers (MISPs). To provide better services, it is quite necessary for MISPs to analyze the information hidden in the big data stream generated by users. Therefore, processing the real-time big data stream efficiently has become increasingly important. However, traditional static data storage technology fails to meet the demands of real-time data processing. To improve processing capacity, many parallel processing structures are proposed, which brings up the problem about how the parallel devices can be scheduled to maximize their efficiency. Accordingly, a dynamic assignment scheduling algorithm for big data stream processing in mobile Internet services is proposed, and a stream query graph is built to calculate the weight of every edge. The edge with the minimum weight is selected to send tuples. Simulation results show that the proper number of the logic devices can dramatically reduce system response time. Furthermore, system context switching is reduced by increasing the number of tuples sent each time.     

Research on K Maximum Dominant Skyline and E-GA Algorithm Based on Data Stream Environment

With the continuous development of database technology, the data volume that can be stored and processed by the database is increasing. How to dig out information that people are interested in from the massive data is one of the important issues in the field of database research. This article starts from the user demand analysis, and makes an in-depth study of various query expansion problems of skylines. Then, according to different application scenarios, this paper proposes efficient and targeted solutions to effectively meet the actual needs of people. Based on k- representative skyline query problem in the data stream environment, a k-representative skyline selection standard k-LDS is presented which is applicable for data stream environment. k-LDS hopes to select the skyline subset with the largest dominant area (containing k skyline tuples only) as k- representative skyline set in data stream. And for the 3-dimensionalal and multidimensional k-LDS problems, this paper also proposes the approximation algorithm, namely GA algorithm. Finally, through the experiment, it is proved that k-LDS is more suitable for the data stream environment, and the algorithm proposed can effectively solve k-LD problems under the data stream environment.     

Load shedding for multi-way stream joins based on arrival order patterns

We address the problem of load shedding for continuous multi-way join queries over multiple data streams. When the arrival rates of tuples from data streams exceed the system capacity, a load shedding algorithm drops some subset of input tuples to avoid system overloads. To decide which tuples to drop among the input tuples, most existing load shedding algorithms determine the priority of each input tuple based on the frequency or some historical statistics of its join attribute value, and then drop tuples with the lowest priority. However, those value-based algorithms cannot determine the priorities of tuples properly in environments where join attribute values are unique and each join attribute value occurs at most once in each data stream. In this paper, we propose a load shedding algorithm specifically designed for such environments. The proposed load shedding algorithm determines the priority of each tuple based on the order of streams in which its join attribute value appears, rather than its join attribute value itself. Consequently, the priorities of tuples can be determined effectively in environments where join attribute values are unique and do not repeat. The experimental results show that the proposed algorithm outperforms the existing algorithms in such environments in terms of effectiveness and efficiency.     

一种面向混合属性数据聚类的新算法

在分析传统聚类算法的基础上,提出一种针对混合属性数据的聚类算法.该算法利用格论中简单元组及超级元组将对象属性转化为格模型建立,以对象间格覆盖数来衡量类间相似度,根据高覆盖数高相似度的原则选择聚类中心进行聚类.在公共数据集上的实验结果表明,该算法在不增加空间复杂度的基础上,有效地提高了混合属性数据聚类的质量.