On entropy-based term weighting schemes for text categorization

In text categorization, Vector Space Model (VSM) has been widely used for representing documents, in which a document is represented by a vector of terms. Since different terms contribute to a document’s semantics in various degrees, a number of term weighting schemes have been proposed for VSM to improve text categorization performance. Much evidence shows that the performance of a term weighting scheme often varies across different text categorization tasks, while the mechanism underlying variability in a scheme’s performance remains unclear. Moreover, existing schemes often weight a term with respect to a category locally, without considering the global distribution of a term’s occurrences across all categories in a corpus. In this paper, we first systematically examine pros and cons of existing term weighting schemes in text categorization and explore the reasons why some schemes with sound theoretical bases, such as chi-square test and information gain, perform poorly in empirical evaluations. By measuring the concentration that a term distributes across all categories in a corpus, we then propose a series of entropy-based term weighting schemes to measure the distinguishing power of a term in text categorization. Through extensive experiments on five different datasets, the proposed term weighting schemes consistently outperform the state-of-the-art schemes. Moreover, our findings shed new light on how to choose and develop an effective term weighting scheme for a specific text categorization task.

     

Supervised and Traditional Term Weighting Methods for Automatic Text Categorization

In vector space model (VSM), text representation is the task of transforming the content of a textual document into a vector in the term space so that the document could be recognized and classified by a computer or a classifier. Different terms (i.e. words, phrases, or any other indexing units used to identify the contents of a text) have different importance in a text. The term weighting methods assign appropriate weights to the terms to improve the performance of text categorization. In this study, we investigate several widely-used unsupervised (traditional) and supervised term weighting methods on benchmark data collections in combination with SVM and kNN algorithms. In consideration of the distribution of relevant documents in the collection, we propose a new simple supervised term weighting method, i.e. tf.rf, to improve the terms' discriminating power for text categorization task. From the controlled experimental results, these supervised term weighting methods have mixed performance. Specifically, our proposed supervised term weighting method, tf.rf, has a consistently better performance than other term weighting methods while other supervised term weighting methods based on information theory or statistical metric perform the worst in all experiments. On the other hand, the popularly used tf.idf method has not shown a uniformly good performance in terms of different data sets.     

Term frequency with average term occurrences for textual information retrieval

In the context of information retrieval (IR) from text documents, the term weighting scheme (TWS) is a key component of the matching mechanism when using the vector space model. In this paper, we propose a new TWS that is based on computing the average term occurrences of terms in documents and it also uses a discriminative approach based on the document centroid vector to remove less significant weights from the documents. We call our approach Term Frequency With Average Term Occurrence (TF-ATO). An analysis of commonly used document collections shows that test collections are not fully judged as achieving that is expensive and maybe infeasible for large collections. A document collection being fully judged means that every document in the collection acts as a relevant document to a specific query or a group of queries. The discriminative approach used in our proposed approach is a heuristic method for improving the IR effectiveness and performance and it has the advantage of not requiring previous knowledge about relevance judgements. We compare the performance of the proposed TF-ATO to the well-known TF-IDF approach and show that using TF-ATO results in better effectiveness in both static and dynamic document collections. In addition, this paper investigates the impact that stop-words removal and our discriminative approach have on TF-IDF and TF-ATO. The results show that both, stop-words removal and the discriminative approach, have a positive effect on both term-weighting schemes. More importantly, it is shown that using the proposed discriminative approach is beneficial for improving IR effectiveness and performance with no information on the relevance judgement for the collection.     

用于文本情感分析的特征加权改进算法

文本情感分析领域内的特征加权一般考虑两个影响因子：特征在文档中的重要性（ITD）和特征在表达情感上的重要性（ITS）。结合该领域内两种分类准确率较高的监督特征加权算法,提出了一种新的ITS算法。新算法同时考虑特征在一类文档集里的文档频率（在特定的文档集里,出现某个特征的文档数量）及其占总文档频率的比例,使主要出现且大量出现在同一类文档集里的特征获得更高的ITS权值。实验证明,新算法能提高文本情感分类的准确率。     

Using the absolute difference of term occurrence probabilities in?binary text categorization

In this study, the differences among widely used weighting schemes are studied by means of ordering terms according to their discriminative abilities using a recently developed framework which expresses term weights in terms of the ratio and absolute difference of term occurrence probabilities. Having observed that the ordering of terms is dependent on the weighting scheme under concern, it is emphasized that this can be explained by the way different schemes use term occurrence differences in generating term weights. Then, it is proposed that the relevance frequency which is shown to provide the best scores on several datasets can be improved by taking into account the way absolute difference values are used in other widely used schemes. Experimental results on two different datasets have shown that improved F ₁ scores can be achieved.     

基于局部特征聚焦的方面级情感分析

现有方面级情感分析模型忽略了各词间句法关系且未能针对性地提取语义信息。为此,提出一种可聚焦局部上下文特征的方面级情感分析模型,其核心思想在于构建局部上下文加权邻接图和动态赋权方法,通过图卷积神经网络生成聚焦于局部上下文信息的方面词特征。具体地,首先采用局部上下文动态赋权方式增加局部上下文的关注度;其次,在提取句法依存关系的基础上为上下文各节点赋权,构建针对局部上下文赋权的邻接图;最后,由图卷积神经网络提取聚焦于局部上下文信息的方面词特征。在公开数据集上的实验结果表明,与ASGCN相比,提出模型在restaurant和laptop数据集中的宏F₁值分别提高了1.76%和1.12%,经过局部上下文加权,聚焦局部特征所得信息有助于提高分类效果。     

基于二维信息增益加权的朴素贝叶斯分类算法

由于朴素贝叶斯算法的特征独立性假设以及传统TFIDF加权算法仅仅考虑了特征在整个训练集的分布情况,忽略了特征与类别和文档之间关系,造成传统方法赋予特征的权重并不能代表其准确性.针对以上问题,提出了二维信息增益加权的朴素贝叶斯分类算法,进一步考虑到了特征的二维信息增益即特征类别信息增益和特征文档信息增益对分类效果的影响,并设计实验与传统的加权朴素贝叶斯算法相比,该算法在查准率、召回率、F1值指标性能上能提升6%左右.     

基于情感词汇与机器学习的方面级情感分类

为综合利用基于情感词典和基于机器学习的两类情感分类方法的优点,提出一种基于情感词汇与机器学习的方面级情感分类方法。通过选取少量情感倾向与评价对象无关的情感词汇对评价搭配进行情感分类;通过构建机器学习分类器,以评价短语对各类别的互信息占比作为分类器的分类概率权重,进行加权计算,选择加权后分类概率最大的类别作为评价搭配的情感倾向类别。在中文评论数据集上的实验结果表明,该方法能有效提高情感分类性能。     

Ranking entities on the basis of users’ opinions

Online opinions are one of the most important sources of information on which users base their purchasing decisions. Unfortunately, the large quantity of opinions makes it difficult for an individual to consume in a reasonable amount of time. Unlike standard information retrieval problems, the task here is to retrieve entities whose relevance is dependent upon other people’s opinions regarding the entities and how well those sentiments match the user’s own preferences. We propose novel techniques that incorporate aspect subjectivity measures into weighting the relevance of opinions of entities based on a user’s query keywords. We calculate these weights using sentiment polarity of terms found proximity close to keywords in opinion text. We have implemented our techniques, and we show that these improve the overall effectiveness of the baseline retrieval task. Our results indicate that on entities with long opinions our techniques can perform as good as state-of-the-art query expansion approaches.     

基于非完备信息系统的评价对象情感聚类

该文利用领域本体对产品评论文本中的评价对象进行抽取和整合,在此基础上,建立产品性能的非完备信息系统,将特征的情感倾向寓于特征的权重计算之中。对非完备信息系统,给出了基于差别矩阵的启发式特征约简方法,通过特征降维处理,达到了减少特征的冗余度和数据稀疏性的目的。对降维后的非完备信息系统采用K-Means聚类算法,实现了评价对象情感聚类。为了验证该文提出方法的有效性,在真实汽车评论文本数据上进行实验, 实验结果表明,在对特征进行一定程度的降维后,仍表现出较好的聚类效果。     

Term weighting for information retrieval based on term’s discrimination power

One of the most important research topics in Information Retrieval is term weighting for document ranking and retrieval, such as TFIDF, BM25, etc. We propose a term weighting method that utilizes past retrieval results consisting of the queries that contain a particular term, retrieval documents, and their relevance judgments. A term’s Discrimination Power(DP) is based on the difference degree of the term’s average weights obtained from between relevant and non-relevant retrieved document sets. The difference based DP performs better compared to ratio based DP introduced in the previous research. Our experimental result shows that a term weighting scheme based on the discrimination power method outperforms a TF*IDF based scheme.     

Clause Weighting Local Search for SAT

This paper investigates the necessary features of an effective clause weighting local search algorithm for propositional satisfiability testing. Using the recent history of clause weighting as evidence, we suggest that the best current algorithms have each discovered the same basic framework, that is, to increase weights on false clauses in local minima and then to periodically normalize these weights using a decay mechanism. Within this framework, we identify two basic classes of algorithm according to whether clause weight updates are performed additively or multiplicatively. Using a state-of-the-art multiplicative algorithm (SAPS) and our own pure additive weighting scheme (PAWS), we constructed an experimental study to isolate the effects of multiplicative in comparison to additive weighting, while controlling other key features of the two approaches, namely, the use of pure versus flat random moves, deterministic versus probabilistic weight smoothing and multiple versus single inclusion of literals in the local search neighbourhood. In addition, we examined the effects of adding a threshold feature to multiplicative weighting that makes it indifferent to similar cost moves. As a result of this investigation, we show that additive weighting can outperform multiplicative weighting on a range of difficult problems, while requiring considerably less effort in terms of parameter tuning. Our examination of the differences between SAPS and PAWS suggests that additive weighting does benefit from the random flat move and deterministic smoothing heuristics, whereas multiplicative weighting would benefit from a deterministic/probabilistic smoothing switch parameter that is set according to the problem instance. We further show that adding a threshold to multiplicative weighting produces a general deterioration in performance, contradicting our earlier conjecture that additive weighting has better performance due to having a larger selection of possible moves. This leads us to explain differences in performance as being mainly caused by the greater emphasis of additive weighting on penalizing clauses with relatively less weight.     

Clustered intrinsic label correlations for multi-label classification

Currently a consensus on multi-label classification is to exploit label correlations for performance improvement. Many approaches build one classifier for each label based on the one-versus-all strategy, and integrate classifiers by enforcing a regularization term on the global weights to exploit label correlations. However, this strategy might be suboptimal since it may be only part of the global weights that support the assumption. This paper proposes clustered intrinsic label correlations for multi-label classification (CILC), which extends traditional support vector machine to the multi-label setting. The predictive function of each classifier consists of two components: one component is the common information among all labels, and the other component is a label-specific one which highly depends on the corresponding label. The label-specific one representing the intrinsic label correlations is regularized by clustered structure assumption. The appealing features of the proposed method are that it separates the common information and the label-specific information of the labels and utilizes clustered structures among labels represented by the label-specific parts. The practical multi-label classification problems can be directly solved by the proposed CILC method, such as text categorization, image annotation and sentiment analysis. Experiments across five data sets validate the effectiveness of CILC, compared with six well-established multi-label classification algorithms.     

基于辅助记忆循环神经网络的视角级情感分析

现有视角级情感分析方法大多数利用视角词信息从句子中提取特征,不能同时利用视角和视角词信息,导致模型性能较低,为此文中提出基于辅助记忆循环神经网络的视角级情感分析方法.首先通过深度双向长短期记忆网络和单词的位置信息构建位置权重记忆,利用注意力机制结合视角词建立视角记忆.再联合位置权重记忆和视角记忆输入多层门循环单元,得到视角情感特征.最后由归一化函数识别情感极性.实验表明,相对基准实验,文中方法在3个公开数据集上的效果更好,该方法是有效的.     

基于短文本情感增强的在线学习者成绩预测方法

当前利用短文本情感信息进行在线学习成绩预测的研究存在以下问题: 1)当前情感分类模型无法有效适应在线学习社区的短文本特征, 分类效果较差; 2)利用短文本情感信息定量预测在线学习成绩的研究在准确性上还有较大的提升空间. 针对以上问题, 本文提出了一种短文本情感增强的成绩预测方法. 首先, 从单词和句子层面建模短文本语义, 并提出基于学习者特征的注意力机制以识别不同学习者的语言表达特点, 得到情感概率分布向量; 其次, 将情感信息与统计、学习行为信息相融合, 并基于长短时记忆网络建模学习者的学习状态; 最后, 基于学习状态预测学习者成绩. 在三种不同类别课程组成的真实数据集上进行了实验, 结果表明本文方法能有效对学习社区短文本进行情感分类, 且能够提升在线学习者成绩预测的准确性. 同时, 结合实例分析说明了情感信息、学习状态与成绩之间的关联.     

基于互信息的文本特征加权方法

特征加权是文本分类中的重要环节,通过考察传统的特征选择函数,发现互信息方法在特征加权过程中表现尤为突出。为了提高互信息方法在特征加权时的性能,加入了词频信息、文档频率信息以及类别相关度因子,提出了一种基于改进的互信息特征加权方法。实验结果表明,该方法比传统的特征加权方法具有更好的分类性能。     

Enhancing passage retrieval in log files by query expansion based on explicit and pseudo relevance feedback

Passage retrieval is usually defined as the task of searching for passages which may contain the answer for a given query. While these approaches are very efficient when dealing with texts, applied to log files (i.e. semi-structured data containing both numerical and symbolic information) they usually provide irrelevant or useless results. Nevertheless one appealing way for improving the results could be to consider query expansions that aim at adding automatically or semi-automatically additional information in the query to improve the reliability and accuracy of the returned results. In this paper, we present a new approach for enhancing the relevancy of queries during a passage retrieval in log files. It is based on two relevance feedback steps. In the first one, we determine the explicit relevance feedback by identifying the context of the requested information within a learning process. The second step is a new kind of pseudo relevance feedback. Based on a novel term weighting measure it aims at assigning a weight to terms according to their relatedness to queries. This measure, called TRQ (Term Relatedness to Query), is used to identify the most relevant expansion terms.The main advantage of our approach is that is can be applied both on log files and documents from general domains. Experiments conducted on real data from logs and documents show that our query expansion protocol enables retrieval of relevant passages.     

Modified frequency-based term weighting schemes for text classification

With the rapid growth of textual content on the Internet, automatic text categorization is a comparatively more effective solution in information organization and knowledge management. Feature selection, one of the basic phases in statistical-based text categorization, crucially depends on the term weighting methods In order to improve the performance of text categorization, this paper proposes four modified frequency-based term weighting schemes namely; mTF, mTFIDF, TFmIDF, and mTFmIDF. The proposed term weighting schemes take the amount of missing terms into account calculating the weight of existing terms. The proposed schemes show the highest performance for a SVM classifier with a micro-average F1 classification performance value of 97%. Moreover, benchmarking results on Reuters-21578, 20Newsgroups, and WebKB text-classification datasets, using different classifying algorithms such as SVM and KNN show that the proposed schemes mTF, mTFIDF, and mTFmIDF outperform other weighting schemes such as TF, TFIDF, and Entropy. Additionally, the statistical significance tests show a significant enhancement of the classification performance based on the modified schemes.     

A new image fusion performance metric based on visual information fidelity

Because subjective evaluation is not adequate for assessing work in an automatic system, using an objective image fusion performance metric is a common approach to evaluate the quality of different fusion schemes. In this paper, a multi-resolution image fusion metric using visual information fidelity (VIF) is presented to assess fusion performance objectively. This method has four stages: (1) Source and fused images are filtered and divided into blocks. (2) Visual information is evaluated with and without distortion information in each block. (3) The visual information fidelity for fusion (VIFF) of each sub-band is calculated. (4) The overall quality measure is determined by weighting the VIFF of each sub-band. In our experiment, the proposed fusion assessment method is compared with several existing fusion metrics using the subjective test dataset provided by Petrovic. We found that VIFF performs better in terms of both human perception matching and computational complexity.     

Chinese document re-ranking based on automatically acquired term resource

In this paper, we address the problem of document re-ranking in information retrieval, which is usually conducted after initial retrieval to improve rankings of relevant documents. To deal with this problem, we propose a method which automatically constructs a term resource specific to the document collection and then applies the resource to document re-ranking. The term resource includes a list of terms extracted from the documents as well as their weighting and correlations computed after initial retrieval. The term weighting based on local and global distribution ensures the re-ranking not sensitive to different choices of pseudo relevance, while the term correlation helps avoid any bias to certain specific concept embedded in queries. Experiments with NTCIR3 data show that the approach can not only improve performance of initial retrieval, but also make significant contribution to standard query expansion.