一种与缓冲区紧耦合的环形循环滑动窗口的数据流抽取算法

 许多应用领域产生数据流的流量不断增大,需要挖掘系统提供更有效的数据流抽取策略来应对数据处理能力与流速之间的矛盾.本文构造了基于环形缓冲区循环的逻辑滑动窗口(环形循环滑动窗口),将环形缓冲区和环形循环滑动窗口紧耦合,提出了基于环形循环滑动窗口的数据流抽取算法.实验表明这是适合数据流流动和方便数据流挖掘的数据流抽取模型,能够提升数据流抽取速度和保证数据流的挖掘持续正常实施.     

分布式数据流上的连续异常检测

近年来,数据流异常检测在决策支持和监测等领域有着广泛的应用前景,并成为数据管理与挖掘的研究热点.针对该问题提出了相应的异常定义及检测算法,理论分析表明:与现有异常检测算法相比较,提出的算法具有良好的性能和效率,更适合于数据流应用.     

基于数据流方法的大规模网络异常发现

随着网络规模和速度的增加,大规模网络异常发现要求检测算法能够在无保留状态或者少保留状态下对G比特级的海量网络业务量数据进行实时在线分析。针对在高速骨干网上进行大规模网络异常发现的特点和要求,提出了一种基于数据流的大规模网络异常发现的方法,第一次将数据流模型用于大规模网络的异常发现。主要包括以下创新点:设计了一种面向异常发现的网络流量概要数据结构和突发高频事件检测算法;提出了一种基于安全监测策略定制的预查询方法来进行多数据流的关联监测并且对数据流查询进行了优化;在真实数据分析的基础上,对网络业务量进行了数据约减,使得监测部分特殊类型的数据流能最大程度地获得整体网络业务量的变化特征以提高异常发现的效率。通过真实网络环境下的实验和性能评价验证了数据流方法的有效性。     

数据流中频繁闭项集的近似挖掘算法

在数据流中挖掘频繁项集得到了广泛的研究,传统的研究方法大多关注于在数据流中挖掘全部频繁项集.由于挖掘全部频繁项集存在数据和模式冗余问题,所以对算法的时间和空间效率都具有更大的挑战性.因此,近年来人们开始关注在数据流中挖掘频繁闭项集,其中一个典型的工作就是Moment算法.本文提出了一种数据流中频繁闭项集的近似挖掘算法A-Moment.它采用衰减窗口机制、近似计数估计方法和分布式更新信息策略来解决Moment算法中过度依赖于窗口和执行效率低等问题.实验表明,该算法在保证挖掘精度的前提下,可以比Moment获得更好的效率.     

基于数据流挖掘算法的电网设备风险识别研究

基于电力设备数据流的复杂特性,首先以双层滑动窗口进行数据的预处理,其次以聚类分析作为数据流挖掘技术,进行电力设备异常数据的分类识别,最后以M电力企业的变压器故障数据作为测试集,将双层窗口的数据流分类风险识别算法、Euclidean cluster算法和LS-SVM算法进行性能对比.从算法计算时间、准确率和有效数据误删除...     

用于电力数据管理分析的负荷预测与异常检测

构建了一个电力数据管理分析系统,并设计了电力负荷预测算法和异常数据检测算法问题.首先,针对BP神经网络在预测电力负荷存在的因初始权值与阈值设置影响估计精度的问题,提出利用粒子群优化BP神经网络网络参数,提高了预测算法的收敛速度与预测精度;然后,针对电力数据异常检测算法效率较低的问题,提出了基于改进谱聚类的异常数据检测算...     

基于改进LOF的高维数据异常检测方法

作为数据挖掘的核心问题之一，检测离群点或异常值是及时发现故障和隐患问题的重要判断依据。随着物联网设备量的持续增长，传统的单维异常检测算法已经难以满足日益复杂的大数据应用场景。对多维、庞大的数据流进行异常检测时，容易发生检测速度慢和研判准确度下降的问题。本文提出了一个基于高维数据的改进LOF异常检测算法，以提高检测速度和检测精度。同时构建了一个面向海量监控指标数据的流式处理框架，保障异常检测的正常运行。实验结果表明，改进后的算法在准确率和计算效率上有明显提升。     

基于改进聚类算法的网络平台异常数据挖掘方法

为提高挖掘结果与对应异常数据类型之间的关联度,确保挖掘结果能够为网络平台异常状态识别提供有力依据,文章引入改进聚类算法,开展网络平台异常数据挖掘方法设计研究.通过基于改进聚类算法的网络平台运行数据分类、网络平台异常检测、网络平台分布式最大频繁序列提取、最大频繁序列数据比对与挖掘,提出一种全新的挖掘方法.通过对比实验结果...     

数据流挖掘研究进展

随着企业信息化程度的提高和互联网的普及,每天都会产生海量的实时数据,而数据流挖掘则为分析海量数据提供了一种新途径.目前,数据流挖掘已成为数据挖掘研究领域一个重要分支.本文介绍了数据流挖掘的研究进展,重点讨论了数据流挖掘中的聚类、分类、离群点检测,并对数据流挖掘未来的研究热点进行了展望.     

基于秩2更新的多维数据流典型相关跟踪算法

 现存的多维数据流典型相关分析(Canonical Correlation Analysis,简称CCA)算法主要是基于近似技术的求解方法,本质上并不是持续更新的精确算法.为了能在时变的环境中持续、快速而精确地跟踪数据流之间的相关性,本文提出一种多维数据流典型相关跟踪算法TCCA.该算法基于秩2更新理论,通过并行方式持续更新样本协方差矩阵的特征子空间,进而实现多维数据流典型相关的快速跟踪.理论分析及仿真实验结果表明,TCCA具有较好的稳定性、较高的计算效率和精度,可以作为基本工具应用于数据流相关性检测、特征融合、数据降维等数据流挖掘领域.     

Efficient Mining of the Multidimensional Traffic Cluster Hierarchy for Digesting, Visualization, and Anomaly Identification

Mining traffic to identify the dominant flows sent over a given link, over a specified time interval, is a valuable capability with applications to traffic auditing, simulation, visualization, as well as anomaly detection. Recently, Estan advanced a comprehensive data mining structure tailored for networking data—a parsimonious, multidimensional flow hierarchy, along with an algorithm for its construction. While they primarily targeted offline auditing, use in interactive traffic visualization and anomaly/attack detection will require real-time data mining. We suggest several improvements to Estan 's algorithm that substantially reduce the computational complexity of multidimensional flow mining. We also propose computational and memory-efficient approaches for unidimensional clustering of the IP address spaces. For baseline implementations, evaluated on the New Zealand (NZIX) trace data, our method reduced CPU execution times of the Estan method by a factor of more than eight. We also develop a methodology for anomaly/attack detection based on flow mining, demonstrating the usefulness of this approach on traces from the Slammer and Code Red worms and the MIT Lincoln Laboratories DDoS data.     

Fast wireless sensor for anomaly detection based on data stream in an edge-computing-enabled smart greenhouse

Edge-computing-enabled smart greenhouses are a representative application of the Internet of Things (IoT) technology, which can monitor the environmental information in real-time and employ the information to contribute to intelligent decision-making. In the process, anomaly detection for wireless sensor data plays an important role. However, the traditional anomaly detection algorithms originally designed for anomaly detection in static data do not properly consider the inherent characteristics of the data stream produced by wireless sensors such as infiniteness, correlations, and concept drift, which may pose a considerable challenge to anomaly detection based on data stream and lead to low detection accuracy and efficiency. First, the data stream is usually generated quickly, which means that the data stream is infinite and enormous. Hence, any traditional off-line anomaly detection algorithm that attempts to store the whole dataset or to scan the dataset multiple times for anomaly detection will run out of memory space. Second, there exist correlations among different data streams, and traditional algorithms hardly consider these correlations. Third, the underlying data generation process or distribution may change over time. Thus, traditional anomaly detection algorithms with no model update will lose their effects. Considering these issues, a novel method (called DLSHiForest) based on Locality-Sensitive Hashing and the time window technique is proposed to solve these problems while achieving accurate and efficient detection. Comprehensive experiments are executed using a real-world agricultural greenhouse dataset to demonstrate the feasibility of our approach. Experimental results show that our proposal is practical for addressing the challenges of traditional anomaly detection while ensuring accuracy and efficiency.     

自适应模糊决策树算法在数据流挖掘中的应用

在网络的许多应用中数据是以流的形式存在的,例如网络流、传感器数据,以及网页点击流等,分析和挖掘这类数据,可以发现某中有价值的信息。在此,针对数据流挖掘算法中出现的一些问题（如概念漂移问题）,提出了一种自适应模糊决策树的优化算法。该算法对于解决处理数据流概念中的漂移问题有较好的效果。     

多核DSP上的ISAR实时成像技术研究

为了提高逆合成孔径雷达（ISAR）实时成像的性能,本文首先设计了一种基于TMS320C6678多核信号处理器（DSP）的高速实时信号处理平台,优化了功耗的同时提高了信号处理能力。其次,本文提出了一种利用窄带测量信息进行成像条件判断、成像数据选择和指导高速运动补偿的实时成像流程,并通过将该流程分割成几个独立的任务,在分析任务的实时性和任务间的通信的基础上,完成了任务在多核DSP上的分配。利用本文平台对实测数据进行处理,并将成像性能和实时性与单核DSP信号处理平台做对比,进一步验证了多核信号处理平台的处理优势和算法设计的合理性。      

基于分布式防火墙日志的入侵检测

采用数据挖掘技术,将误用检测和异常检测相结合,对分布式防火墙日志进行入侵检测分析.实验数据及分析结果表明,通过将两种入侵检测方法相结合的方式对入侵行为具有较高检测率和较低的误报率,具有一定的实际应用意义.     

一种动态视频流实时多人脸检测算法

针对传统的AdaBoost人脸检测算法对侧面及多姿态人脸检测误检率较高且检测速度较慢等问题,提出了一种动态视频流实时多人脸检测算法。仿真实验表明,通过使用该算法对静态图像以及动态视频流中的多个人脸实时检测,其结果比传统算法具有更低的误检率和更快检测速度。     

Scheduling algorithm for stochastic job stream cipher service based on load balancing

Business stream parallel processing system face the situation of the diversity of cipher service requests,the cross of serial mode and parallel mode,the intercross of different business data flow,and the demand of high speed and high reliability in security field.In order to improve the parallel processing efficiency of multi-cryptographic algorithm,load balancing was used as scheduling objective.Based on hierarchical hardware scheduling method (HHS-ACDID),considering the processing node’s storage capacity and processing speed,a load balancing scheduling algorithm was approved to support non related tasks and related tasks at the same time,which achieves the high speed cipher processing throughput.Simulation results show that the algorithm can complete dynamic scheduling of data stream system and get better load balancing effect.Compared with HHS-ACDID,the efficiency of the algorithm is improved by about 12%.     

A VLSI Architecture and Algorithm for Lucas–Kanade-Based Optical Flow Computation

Optical flow computation in vision-based systems demands substantial computational power and storage area. Hence, to enable real-time processing at high resolution, the design of application-specific system for optic flow becomes essential. In this paper, we propose an efficient VLSI architecture for the accurate computation of the Lucas–Kanade (L-K)-based optical flow. The L-K algorithm is first converted to a scaled fixed-point version, with optimal bit widths, for improving the feasibility of high-speed hardware implementation without much loss in accuracy. The algorithm is mapped onto an efficient VLSI architecture and the data flow exploits the principles of pipelining and parallelism. The optical flow estimation involves several tasks such as Gaussian smoothing, gradient computation, least square matrix calculation, and velocity estimation, which are processed in a pipelined fashion. The proposed architecture was simulated and verified by synthesizing onto a Xilinx Field Programmable Gate Array, which utilize less than 40% of system resources while operating at a frequency of 55 MHz. Experimental results on benchmark sequences indicate 42% improvement in accuracy and a speed up of five times, compared to a recent hardware implementation of the L-K algorithm.