An efficient sampling-based visualization technique for big data clustering with crisp partitions

The data cluster tendency is an emerging need for exploring the big data cluster analysis tasks. The data are evaluated based on the number of clusters is known as cluster tendency. Many visualization techniques have been developed for the detection of cluster tendency. Some of the existing techniques include Visual Assessment Tendency (VAT), spectral-based VAT (SpecVAT), and improved VAT (iVAT), are considerably succeeded for an assessment of cluster tendency for small datasets. A bigVAT is another method that was recently developed for the estimation of cluster tendency of big data. It is perfect for deriving the clustering tendency in visual form for big data. However, it is intractable to explore the data clusters for large volumes of data objects. The proposed work addresses the clustering problem of bigVAT with the derivation of sampling-based crisp partitions. The crisp partitions will accurately predict the cluster labels of data objects. This research is based on big synthetic and big real-life datasets for demonstrating the performance efficiency of the proposed work.

     

Scalable visual assessment of cluster tendency for large data sets

The problem of determining whether clusters are present in a data set (i.e., assessment of cluster tendency) is an important first step in cluster analysis. The visual assessment of cluster tendency (VAT) tool has been successful in determining potential cluster structure of various data sets, but it can be computationally expensive for large data sets. In this article, we present a new scalable, sample-based version of VAT, which is feasible for large data sets. We include analysis and numerical examples that demonstrate the new scalable VAT algorithm.     

一种基于符号关系图的快速符号数据聚类算法

由于在实际应用中有大量的符号数据生成,符号数据聚类成为了聚类分析的一个重要研究领域。目前,已有许多符号数据聚类算法被提出,但将它们应用于大数据环境时,仍然存在计算成本高、运行速度慢等问题。文中提出了一种基于符号关系图的快速符号数据聚类算法。该算法使用符号关系图替代原始数据,缩小数据集的规模,有效地解决了这一问题。大量的实验分析显示新算法相比其他算法是有效的。     

Automatically Determining the Number of Clusters in Unlabeled Data Sets

Clustering is a popular tool for exploratory data analysis. One of the major problems in cluster analysis is the determination of the number of clusters in unlabeled data, which is a basic input for most clustering algorithms. In this paper we investigate a new method called DBE (dark block extraction) for automatically estimating the number of clusters in unlabeled data sets, which is based on an existing algorithm for visual assessment of cluster tendency (VAT) of a data set, using several common image and signal processing techniques. Basic steps include: 1) generating a VAT image of an input dissimilarity matrix; 2) performing image segmentation on the VAT image to obtain a binary image, followed by directional morphological filtering; 3) applying a distance transform to the filtered binary image and projecting the pixel values onto the main diagonal axis of the image to form a projection signal; 4) smoothing the projection signal, computing its first-order derivative, and then detecting major peaks and valleys in the resulting signal to decide the number of clusters. Our new DBE method is nearly "automatic", depending on just one easy-to-set parameter. Several numerical and real-world examples are presented to illustrate the effectiveness of DBE.     

Visualization of balancing systems based on naïve psychological approaches

In this paper, we propose a novel medium for interactions based on an interpersonal psychological approach referred to as ‘naïve psychology’. We adopt the visual assessment of clustering tendency (VAT) to naïve psychology for the visual understanding of other people. The VAT algorithm produces a visual display that can be used to assess clustering tendencies in a set of persons (notions) by reconstructing a digital image representation of a square relational dissimilarity matrix for its set. This algorithm clearly represents two types of imbalanced situations in naïve psychology: crisp and fuzzy. The visual image of a balanced or imbalance situation is useful for a deeper human understanding.     

Visual Assessment of Clustering Tendency for Rectangular Dissimilarity Matrices

We have an m times n matrix D, and assume that its entries correspond to pair wise dissimilarities between m row objects O_r and n column objects O_c, which, taken together (as a union), comprise a set O of N = m + n objects. This paper develops a new visual approach that applies to four different cluster assessment problems associated with O. The problems are the assessment of cluster tendency: PI) amongst the row objects O_r; P2) amongst the column objects O_c; P3) amongst the union of the row and column objects O_r U O_c; and P4) amongst the union of the row and column objects that contain at least one object of each type (co-clusters). The basis of the method is to regard D as a subset of known values that is part of a larger, unknown N times N dissimilarity matrix, and then impute the missing values from D. This results in estimates for three square matrices (D_r, D_c, D_rUc) that can be visually assessed for clustering tendency using the previous VAT or sVAT algorithms. The output from assessment of D_rUc ultimately leads to a rectangular coVAT image which exhibits clustering tendencies in D. Five examples are given to illustrate the new method. Two important points: i) because VAT is scalable by sVAT to data sets of arbitrary size, and because coVAT depends explicitly (and only) on VAT, this new approach is immediately scalable to, say, the scoVAT model, which works for even very large (unloadable) data sets without alteration; and ii) VAT, sVAT and coVAT are autonomous, parameter free models - no "hidden values" are needed to make them work.     

GO: A cluster algorithm for graph visualization

As we are in the big data age, graph data such as user networks in Facebook and Flickr becomes large. How to reduce the visual complexity of a graph layout is a challenging problem. Clustering graphs is regarded as one of effective ways to address this problem. Most of current graph visualization systems, however, directly use existing clustering algorithms that are not originally developed for the visualization purpose. For graph visualization, a clustering algorithm should meet specific requirements such as the sufficient size of clusters, and automatic determination of the number of clusters. After identifying the requirements of clustering graphs for visualization, in this paper we present a new clustering algorithm that is particularly designed for visualization so as to reduce the visual complexity of a layout, together with a strategy for improving the scalability of our algorithm. Experiments have demonstrated that our proposed algorithm is capable of detecting clusters in a way that is required in graph visualization.     

区间值属性数据集关联规则挖掘算法仿真

对区间值属性数据集进行挖掘,可以有效分析出数据之间的关系。针对现有数据挖掘方法未对大规模数据进行聚类,导致挖掘过程占据内存大,挖掘精度低的问题,提出了一种新的区间值属性数据集挖掘算法。对问题定义、数据准备、数据提取、模式预测和数据聚类等模块进行详细分析,完成区间值属性数据聚类。根据聚类结果,将区间值属性数据分成多个数据集,挑选出能够支持最小支持度的项目集,将这些项目集作为频繁项集,进而提取出数据集之间的关联规则,将关联规则融入数据计算步骤,完成数据挖掘。为验证算法效果,进行仿真,结果表明,相较于传统挖掘算法,所提挖掘算法占用容量更小,挖掘精度更高。     

Distributed data clustering in sensor networks

Low overhead analysis of large distributed data sets is necessary for current data centers and for future sensor networks. In such systems, each node holds some data value, e.g., a local sensor read, and a concise picture of the global system state needs to be obtained. In resource-constrained environments like sensor networks, this needs to be done without collecting all the data at any location, i.e., in a distributed manner. To this end, we address the distributed clustering problem, in which numerous interconnected nodes compute a clustering of their data, i.e., partition these values into multiple clusters, and describe each cluster concisely. We present a generic algorithm that solves the distributed clustering problem and may be implemented in various topologies, using different clustering types. For example, the generic algorithm can be instantiated to cluster values according to distance, targeting the same problem as the famous k-means clustering algorithm. However, the distance criterion is often not sufficient to provide good clustering results. We present an instantiation of the generic algorithm that describes the values as a Gaussian Mixture (a set of weighted normal distributions), and uses machine learning tools for clustering decisions. Simulations show the robustness, speed and scalability of this algorithm. We prove that any implementation of the generic algorithm converges over any connected topology, clustering criterion and cluster representation, in fully asynchronous settings.     

Cluster center initialization algorithm for K-modes clustering

Partitional clustering of categorical data is normally performed by using K-modes clustering algorithm, which works well for large datasets. Even though the design and implementation of K-modes algorithm is simple and efficient, it has the pitfall of randomly choosing the initial cluster centers for invoking every new execution that may lead to non-repeatable clustering results. This paper addresses the randomized center initialization problem of K-modes algorithm by proposing a cluster center initialization algorithm. The proposed algorithm performs multiple clustering of the data based on attribute values in different attributes and yields deterministic modes that are to be used as initial cluster centers. In the paper, we propose a new method for selecting the most relevant attributes, namely Prominent attributes, compare it with another existing method to find Significant attributes for unsupervised learning, and perform multiple clustering of data to find initial cluster centers. The proposed algorithm ensures fixed initial cluster centers and thus repeatable clustering results. The worst-case time complexity of the proposed algorithm is log-linear to the number of data objects. We evaluate the proposed algorithm on several categorical datasets and compared it against random initialization and two other initialization methods, and show that the proposed method performs better in terms of accuracy and time complexity. The initial cluster centers computed by the proposed approach are close to the actual cluster centers of the different data we tested, which leads to faster convergence of K-modes clustering algorithm in conjunction to better clustering results.     

“Best K”: critical clustering structures in categorical datasets

The demand on cluster analysis for categorical data continues to grow over the last decade. A well-known problem in categorical clustering is to determine the best K number of clusters. Although several categorical clustering algorithms have been developed, surprisingly, none has satisfactorily addressed the problem of best K for categorical clustering. Since categorical data does not have an inherent distance function as the similarity measure, traditional cluster validation techniques based on geometric shapes and density distributions are not appropriate for categorical data. In this paper, we study the entropy property between the clustering results of categorical data with different K number of clusters, and propose the BKPlot method to address the three important cluster validation problems: (1) How can we determine whether there is significant clustering structure in a categorical dataset? (2) If there is significant clustering structure, what is the set of candidate “best Ks”? (3) If the dataset is large, how can we efficiently and reliably determine the best Ks?     

包空间多示例图像自动分类

为了有效地解决多示例图像自动分类问题,提出一种将多示例图像转化为包空间的单示例描述方法.该方法将图像视为包,图像中的区域视为包中的示例,根据具有相同视觉区域的样本都会聚集成一簇,用聚类算法为每类图像确定其特有的“视觉词汇”,并利用负包示例标注确定的这一信息指导典型“视觉词汇”的选择;然后根据得到的“视觉词汇”构造一个新的空间—包空间,利用基于视觉词汇定义的非线性函数将多个示例描述的图像映射到包空间的一个点,变为单示例描述;最后利用标准的支持向量机进行监督学习,实现图像自动分类.在Corel图像库的图像数据集上进行对比实验,实验结果表明该算法具有良好的图像分类性能.     

A symmetry based multiobjective clustering technique for automatic evolution of clusters

In this paper the problem of automatic clustering a data set is posed as solving a multiobjective optimization (MOO) problem, optimizing a set of cluster validity indices simultaneously. The proposed multiobjective clustering technique utilizes a recently developed simulated annealing based multiobjective optimization method as the underlying optimization strategy. Here variable number of cluster centers is encoded in the string. The number of clusters present in different strings varies over a range. The points are assigned to different clusters based on the newly developed point symmetry based distance rather than the existing Euclidean distance. Two cluster validity indices, one based on the Euclidean distance, XB-index, and another recently developed point symmetry distance based cluster validity index, Sym-index, are optimized simultaneously in order to determine the appropriate number of clusters present in a data set. Thus the proposed clustering technique is able to detect both the proper number of clusters and the appropriate partitioning from data sets either having hyperspherical clusters or having point symmetric clusters. A new semi-supervised method is also proposed in the present paper to select a single solution from the final Pareto optimal front of the proposed multiobjective clustering technique. The efficacy of the proposed algorithm is shown for seven artificial data sets and six real-life data sets of varying complexities. Results are also compared with those obtained by another multiobjective clustering technique, MOCK, two single objective genetic algorithm based automatic clustering techniques, VGAPS clustering and GCUK clustering.     

基于数据集压缩的聚类算法性能优化研究

针对目前聚类算法对大数据集的聚类分析中存在时间花费过大的问题,提出了一种基于最近邻相似性的数据集压缩算法。通过将若干个相似性最近邻的数据点划分成一个数据簇并随机选择簇头构成新的数据集,大大缩减了数据的规模。然后分别采用k-means算法和AP算法对压缩后的数据集进行聚类分析。实验结果表明,压缩后的数据集与原始数据集的聚类分析相比,在保证聚类准确率基本一致的前提下有效降低了聚类的花费时长,提高了算法的聚类性能,证明该数据集压缩算法在聚类分析中的有效性与可靠性。     

一种连通非加权图的快速聚类方法

图的聚类是数据聚类的一种很重要的变体,一方面通常可以用图来表示数据集中数据的相似度;另一方面对大型复杂网络的分析也引起人们越来越多地关注;而且对图进行聚类分析可以增强图的可视性,有助于可视化的分析、观测和导航。将最大最小方法的基本思想应用于非加权图的聚类,提出一种无向连通非加权图的快速聚类方法,该方法具有简单、聚类时间短、运行效率高、对于大型静态图的聚类具有良好的适应性等特点。     

Ensemble-based clustering of large probabilistic graphs using neighborhood and distance metric learning

Graphs are commonly used to express the communication of various data. Faced with uncertain data, we have probabilistic graphs. As a fundamental problem of such graphs, clustering has many applications in analyzing uncertain data. In this paper, we propose a novel method based on ensemble clustering for large probabilistic graphs. To generate ensemble clusters, we develop a set of probable possible worlds of the initial probabilistic graph. Then, we present a probabilistic co-association matrix as a consensus function to integrate base clustering results. It relies on co-occurrences of node pairs based on the probability of the corresponding common cluster graphs. Also, we apply two improvements in the steps before and after of ensembles generation. In the before step, we append neighborhood information based on node features to the initial graph to achieve a more accurate estimation of the probability between the nodes. In the after step, we use supervised metric learning-based Mahalanobis distance to automatically learn a metric from ensemble clusters. It aims to gain crucial features of the base clustering results. We evaluate our work using five real-world datasets and three clustering evaluation metrics, namely the Dunn index, Davies–Bouldin index, and Silhouette coefficient. The results show the impressive performance of clustering large probabilistic graphs.

     

Median graph computation for graph clustering

In this paper, we are interested in the problem of graph clustering. We propose a new algorithm for computing the median of a set of graphs. The concept of median allows the extension of conventional algorithms such as the k-means to graph clustering, helping to bridge the gap between statistical and structural approaches to pattern recognition. Experimental results show the efficiency of the new median graph algorithm compared to the (only) existing algorithm in the literature. We also show its effective use in clustering a set of random graphs and in a content-based synthetic image retrieval system.     

基于核的自适应聚类及其在入侵检测中的应用

针对k-medoid算法不能有效聚类大数据集和高维数据的弱点,将核学习方法引入到k-medoid算法,提出了基于核的自适应k-medoid算法,使其能够对大数据集和高维数据进行聚类.用KDD 99标准数据集进行实验研究,结果表明该算法性能是优良的,并且能获得令人满意的检测效果.