共查询到20条相似文献,搜索用时 234 毫秒
1.
并行化的半监督朴素贝叶斯分类算法 总被引:1,自引:0,他引:1
针对当前需要对海量的文本数据进行分类和用于训练的带标记的文本数据非常匮乏这两个问题,结合半监督的朴素贝叶斯分类算法和Map-Reduce编程模型,提出了一种新型的并行化的半监督朴素贝叶斯分类(parallelized semi-supervised Nave Bayes,PSNB)算法。通过实验可以看出,PSNB算法不仅可以高效地处理海量的文本数据,还可以有效地利用无标记的文本数据来提高分类器准确率。 相似文献
2.
3.
目前如何对互联网上的海量数据进行文本分类已经成为一个重要的研究方向,随着云计算技术和Hadoop平台的逐步发展,文本分类的并行化方式将能够更有效的解决当前的问题.论文针对文本分类中特征选择阶段对文本分类性能有很大影响的缺点,提出了一种改进的特征选择算法——类别相关度算法(Class Correlation Algorithm,CCA),同时根据Hadoop平台在海量数据存储和处理方面所具有的优点,利用MapReduce的并行编程框架和HDFS分布式存储系统对文本分类的各个阶段实现了并行化编程.最后通过实验将Hadoop平台下的文本分类的优化算法与传统的单机运行环境下的文本分类算法进行了对比分析,实验结果表明对于相同的数据集,该算法在运算时间上有极大的提高. 相似文献
4.
5.
袁小艳 《计算机测量与控制》2016,24(1):252-254, 258
随着数据的海量增长,数据聚类算法的研究面临着海量数据挖掘和处理的挑战;针对K-means聚类算法对初始聚类中心的依赖性太强、全局搜索能力也差等缺点,将一种改进的人工蜂群算法与K-means算法相结合,提出了ABC_Kmeans聚类算法,以提高聚类的性能;为了提高聚类算法处理海量数据的能力,采用MapReduce模型对ABC_Kmeans进行并行化处理,分别设计了Map、Combine和Reduce函数;通过在多个海量数据集上进行实验,表明ABC_Kmeans算法的并行化设计具有良好的加速比和扩展性,适用于当今海量数据的挖掘和处理。 相似文献
6.
一种基于Rough Set的海量数据分割算法 总被引:2,自引:0,他引:2
处理海量数据一直是数据挖掘要解决的一个重要问题.目前已有许多并行或串行的算法来处理海量数据,然而这些算法通常都不能很好地解决速度和正确率之间的矛盾.分布式运算在处理数据上具有明显优势,因此本文考虑将一个原始的海量数据集分割成许多个独立的小数据集进行分布式处理.本文首先根据Rough Set的特点提出最佳分割的定义,然后提出一种海量数据分割算法来寻找最佳分割.通过实验测试证明结合本文提出的数据分割算法的分布式处理方案能够快速处理海量数据,而且与处理整个数据集的算法相比,正确性较高. 相似文献
7.
研究朴素贝叶斯算法MapReduce的并行实现方法, 针对传统单点串行算法在面对大规模数据或者参与分类的属性较多时效率低甚至无力承载大规模运算, 以及难以满足人们处理海量数据的需求等问题, 本文在朴素贝叶斯基本理论和MapReduce框架的基础上, 提出了一种基于MapReduce的高效、廉价的并行化方法. 通过实验表明这种方法在面对大规模数据时能有效提高算法的效率, 满足人们处理海量数据的需求. 相似文献
8.
9.
建模连续视觉特征的图像语义标注方法 总被引:1,自引:0,他引:1
针对图像检索中存在的"语义鸿沟"问题,提出一种对连续视觉特征直接建模的图像自动标注方法.首先对概率潜语义分析(PLSA)模型进行改进,使之能处理连续量,并推导对应的期望最大化算法来确定模型参数;然后根据不同模态数据各自的特点,提出一个对不同模态数据分别处理的图像语义标注模型,该模型使用连续PLSA建模视觉特征,使用标准PLSA建模文本关键词,并通过不对称的学习方法学习2种模态之间的关联,从而能较好地对未知图像进行标注.通过在一个包含5000幅图像的标准Corel数据集中进行实验,并与几种典型的图像标注方法进行比较的结果表明,文中方法具有更高的精度和更好的效果. 相似文献
10.
11.
Yun YE Shengrong GONG Chunping LIU Jia ZENG Ning JIA Yi ZHANG 《Frontiers of Computer Science》2013,7(4):526-535
Probabilistic latent semantic analysis (PLSA) is a topic model for text documents, which has been widely used in text mining, computer vision, computational biology and so on. For batch PLSA inference algorithms, the required memory size grows linearly with the data size, and handling massive data streams is very difficult. To process big data streams, we propose an online belief propagation (OBP) algorithm based on the improved factor graph representation for PLSA. The factor graph of PLSA facilitates the classic belief propagation (BP) algorithm. Furthermore, OBP splits the data stream into a set of small segments, and uses the estimated parameters of previous segments to calculate the gradient descent of the current segment. Because OBP removes each segment from memory after processing, it is memory-efficient for big data streams. We examine the performance of OBP on four document data sets, and demonstrate that OBP is competitive in both speed and accuracy for online expectation maximization (OEM) in PLSA, and can also give a more accurate topic evolution. Experiments on massive data streams from Baidu further confirm the effectiveness of the OBP algorithm. 相似文献
12.
Laurence A. F. Park Kotagiri Ramamohanarao 《The VLDB Journal The International Journal on Very Large Data Bases》2009,18(1):141-155
Probabilistic latent semantic analysis (PLSA) is a method for computing term and document relationships from a document set.
The probabilistic latent semantic index (PLSI) has been used to store PLSA information, but unfortunately the PLSI uses excessive
storage space relative to a simple term frequency index, which causes lengthy query times. To overcome the storage and speed
problems of PLSI, we introduce the probabilistic latent semantic thesaurus (PLST); an efficient and effective method of storing
the PLSA information. We show that through methods such as document thresholding and term pruning, we are able to maintain
the high precision results found using PLSA while using a very small percent (0.15%) of the storage space of PLSI. 相似文献
13.
14.
传统潜在语义分析(Latent Semantic Analysis, LSA)方法无法获得场景目标空间分布信息和潜在主题的判别信息。针对这一问题提出了一种基于多尺度空间判别性概率潜在语义分析(Probabilistic Latent Semantic Analysis, PLSA)的场景分类方法。首先通过空间金字塔方法对图像进行空间多尺度划分获得图像空间信息,结合PLSA模型获得每个局部块的潜在语义信息;然后串接每个特定局部块中的语义信息得到图像多尺度空间潜在语义信息;最后结合提出的权值学习方法来学习不同图像主题间的判别信息,从而得到图像的多尺度空间判别性潜在语义信息,并将学习到的权值信息嵌入支持向量基(Support Vector Machine, SVM)分类器中完成图像的场景分类。在常用的三个场景图像库(Scene-13、Scene-15和Caltech-101)上的实验表明,该方法平均分类精度比现有许多state-of-art方法均优。验证了其有效性和鲁棒性。 相似文献
15.
基于PLSA主题模型的多标记文本分类 总被引:1,自引:1,他引:0
为解决多标记文本分类时文本标记关系不明确以及特征维数
过大的问题,提出了基于概率隐语义分析(Probabilistic latent semantic analysis,PL
SA)模型的多标记假设重用文本分类算法。该方法首先将训练样本通过PLSA模型映射到隐语
义空间,以文本的主题分布表示一篇文本,在去噪的同时可以大大降低数据维度。在此基础
上利用多标记假设重用算法(Multi label algorithm of hypothesis reuse,MAHR)进行
分类,由于经过PLSA降维后的特征组本身就具有语义信息,因此算法能够精确地挖掘出多标
记之间的关系并用于训练基分类器,从而避免了人为输入标记关系的缺陷。实验验证了该方
法能够充分利用PLSA降维得到的语义信息来改善多标记文本分类的性能。 相似文献
16.
Zhixin Li Zhongzhi Shi Weizhong Zhao Zhiqing Li Zhenjun Tang 《Engineering Applications of Artificial Intelligence》2013,26(9):2143-2152
Semantic gap has become a bottleneck of content-based image retrieval in recent years. In order to bridge the gap and improve the retrieval performance, automatic image annotation has emerged as a crucial problem. In this paper, a hybrid approach is proposed to learn the semantic concepts of images automatically. Firstly, we present continuous probabilistic latent semantic analysis (PLSA) and derive its corresponding Expectation–Maximization (EM) algorithm. Continuous PLSA assumes that elements are sampled from a multivariate Gaussian distribution given a latent aspect, instead of a multinomial one in traditional PLSA. Furthermore, we propose a hybrid framework which employs continuous PLSA to model visual features of images in generative learning stage and uses ensembles of classifier chains to classify the multi-label data in discriminative learning stage. Therefore, the framework can learn the correlations between features as well as the correlations between words. Since the hybrid approach combines the advantages of generative and discriminative learning, it can predict semantic annotation precisely for unseen images. Finally, we conduct the experiments on three baseline datasets and the results show that our approach outperforms many state-of-the-art approaches. 相似文献
17.
Automatic image annotation has become an important and challenging problem due to the existence of semantic gap. In this paper, we firstly extend probabilistic latent semantic analysis (PLSA) to model continuous quantity. In addition, corresponding Expectation-Maximization (EM) algorithm is derived to determine the model parameters. Furthermore, in order to deal with the data of different modalities in terms of their characteristics, we present a semantic annotation model which employs continuous PLSA and standard PLSA to model visual features and textual words respectively. The model learns the correlation between these two modalities by an asymmetric learning approach and then it can predict semantic annotation precisely for unseen images. Finally, we compare our approach with several state-of-the-art approaches on the Corel5k and Corel30k datasets. The experiment results show that our approach performs more effectively and accurately. 相似文献
18.
在电子商务应用中,为了更好地了解用户的内在特征,制定有效的营销策略,提出一种基于混合概率潜在语义分析(H PLSA)模型的Web聚类算法。利用概率潜在语义分析(PLSA)技术分别对用户浏览数据、页面内容信息及内容增强型用户事务数据建立PLSA模型, 通过对数—似然函数对三个PLSA模型进行合并得到用户聚类的H PLSA模型和页面聚类的H PLSA模型。聚类分析中以潜在主题与用户、页面以及站点之间的条件概率作为相似度计算依据,聚类算法采用基于距离的k medoids 算法。设计并构建了H PLSA模型,在该模型上对Web聚类算法进行验证,表明该算法是可行的。 相似文献
19.
Adaptive Bayesian Latent Semantic Analysis 总被引:1,自引:0,他引:1
Jen-Tzung Chien Meng-Sung Wu 《IEEE transactions on audio, speech, and language processing》2008,16(1):198-207
Due to the vast growth of data collections, the statistical document modeling has become increasingly important in language processing areas. Probabilistic latent semantic analysis (PLSA) is a popular approach whereby the semantics and statistics can be effectively captured for modeling. However, PLSA is highly sensitive to task domain, which is continuously changing in real-world documents. In this paper, a novel Bayesian PLSA framework is presented. We focus on exploiting the incremental learning algorithm for solving the updating problem of new domain articles. This algorithm is developed to improve document modeling by incrementally extracting up-to-date latent semantic information to match the changing domains at run time. By adequately representing the priors of PLSA parameters using Dirichlet densities, the posterior densities belong to the same distribution so that a reproducible prior/posterior mechanism is activated for incremental learning from constantly accumulated documents. An incremental PLSA algorithm is constructed to accomplish the parameter estimation as well as the hyperparameter updating. Compared to standard PLSA using maximum likelihood estimate, the proposed approach is capable of performing dynamic document indexing and modeling. We also present the maximum a posteriori PLSA for corrective training. Experiments on information retrieval and document categorization demonstrate the superiority of using Bayesian PLSA methods. 相似文献
20.
为了能准确挖掘用户兴趣点,首先利用概率潜在语义分析PLSA模型将“网页 词”矩阵向量投影到概率潜在语义向量空间,并提出“自动相似度阈值选择”方法得到网页间的相似度阈值,最后提出将平面划分法与凝聚式层次聚类相结合的凝聚式层次k中心点HAK medoids算法,实现用户兴趣点聚类。实验结果表明,与传统的基于划分的算法相比,HAK medoids算法聚类效果更好。同时,提出的用户兴趣点聚类技术在个性化服务领域可提高个性化推荐和搜索的效率。关键词: 相似文献