A segment-based framework for modeling and mining data streams |
| |
Authors: | Charu C Aggarwal |
| |
Affiliation: | 1. IBM T. J. Watson Research Center, 19 Skyline Drive, Hawthorne, NY, 10532, USA
|
| |
Abstract: | Data Streams have become ubiquitous in recent years because of advances in hardware technology which have enabled automated
recording of large amounts of data. The primary constraint in the effective mining of streams is the large volume of data
which must be processed in real time. In many cases, it is desirable to store a summary of the data stream segments in order
to perform data mining tasks. Since density estimation provides a comprehensive overview of the probabilistic data distribution
of a stream segment, it is a natural choice for this purpose. A direct use of density distributions can however turn out to
be an inefficient storage and processing mechanism in practice. In this paper, we introduce the concept of cluster histograms, which provides an efficient way to estimate and summarize the most important data distribution profiles over different stream
segments. These profiles can be constructed in a supervised or unsupervised way depending upon the nature of the underlying
application. The profiles can also be used for change detection, anomaly detection, segmental nearest neighbor search, or
supervised stream segment classification. Furthermore, these techniques can also be used for modeling other kinds of data
such as text and categorical data. The flexibility of the tasks which can be performed from the cluster histogram framework
follows from its generality in storing the historical density profile of the data stream. As a result, this method provides a holistic framework for density-based mining of data streams. We discuss
and test the application of the cluster histogram framework to a variety of interesting data mining applications. |
| |
Keywords: | |
本文献已被 SpringerLink 等数据库收录! |
|