基于连通域的汉字切分技术研究

字符切分技术已经成为汉字识别系统设计中的关键问题,对于质量较差的文本图像,用灰度图像取代传统的二值化黑白图像能够取得更好的切分效果,基于连通域的切分算法能够对灰度图像进行较好的切分,基于连通域的汉字切分算法能有效地对文本图像中汉字字符部件进行合并及对粘连字符进行分割。     

基于多阈值和多切分策略的间隙切分算法

在分析传统手写汉字切分算法的基础上,提出了基于多阈值和多切分策略的间隙切分算法。算法初切分阶段就字符间距通过多个阈值进行切分,从而得到预切分的最佳结果,在二次切分阶段,通过多个阈值对初切分阶段的切分字块进行再次切分。实验结果表明,该算法相对于一般的间隙切分法,对汉字切分的准确率有较大的提高。     

串频统计和词形匹配相结合的汉语自动分词系统

本文介绍了一种汉语自动分词软件系统,该系统对原文进行三遍扫描:第一遍,利用切分标记将文本切分成汉字短串的序列;第二遍,根据各短串的每个子串在上下文中的频度计算其权值,权值大的子串视为候选词;第三遍,利用候选词集和一部常用词词典对汉字短串进行切分。实验表明,该分词系统的分词精度在1.5%左右,能够识别大部分生词,特别适用于文献检索等领域。     

手写中文信封的地址行字符切分算法

在手写体中文信封处理系统中,地址行字符切分是实现地址行识别的关键步骤。本文根据邮政信封地址行字符的特点,有针对性的提出了一种字符切分算法。首先对地址行图像利用投影、求连通区域、笔划穿越数分析等基于字符结构的方法进行初始切分,得到基本字段序列;然后通过对相邻的基本字段进行组合形成多条候选切分路径,再通过识别的可信度和邮政目标地址库的先验知识信息对路径进行评价分析,从而得到最优的切分路径。该算法经过邮政分拣机采集的实际信封图像测试,纯地址行识别正确率达到78.61%,地址行识别与邮政编码识别相结合的分拣正确率达到95.42%。     

基于凸包像素比特征的粘连汉字切分

汉字切分正确与否直接影响了汉字识别系统的识别率,粘连汉字则是切分中的难点。本文将基于背景细化的切分方法应用于《四库全书》的两字符粘连汉字数据集,并针对其中切分路径选择问题,提出了一种新特征- 凸包像素比,反映了在不同切分路径下汉字结构变化的特性。实验结果表明该特征对多种分类器均能有效地提高切分路径选择的正确率。其中在使用基于高斯混合模型分类器时取得了8816 %正确率。     

用于粘连手写体汉字切分的二次寻点算法

离线手写体汉字的切分是离线汉字识别的基础,其中以粘连字体的切分最难。文章提出了一种对粘连手写体汉字粘连位置精确定位的算法。该算法包含两次寻点过程,第一步寻找波谷点,对粘连字的分割位置进行粗定位;第二步寻找理想切分点,即结合波谷点与波谷邻近特征点的综合位置,最终给出切分线定位,以实现粘连汉字的切分。     

基于候选特征笔画和多类阈值的手写汉字切分

通过分析汉字的常见结构,鉴于汉字与汉字之间的距离和构成汉字的部件之间的距离的显著差异性,提出一种基于候选特征笔画和多类阈值的手写汉字切分方法.首先从构成手写汉字的笔画集合中提取候选特征笔画,根据候选特征笔画将手写汉字预切分,然后利用基于间距阈值的部件组合规则对过切分的汉字部件进行组合,最后采用基于单字宽度阈值的粘连汉字判断规则搜索粘连汉字,对粘连汉字进行递归切分.实验表明,该方法对连续手写汉字的切分准确率较高,具有一定的实用性.     

MST在手写汉字切分中的应用

手写汉字切分是根据输入笔迹的空间位置关系进行汉字部件的合并切分,形成完整的汉字笔划以便进行识别处理.综合利用了汉字部件的结构位置关系和笔划的空间位置关系,根据笔划的最小生成树(minimal spanningtree,简称MST)对联机连续手写输入汉字进行切分,取得了较好的切分结果.切分的准确率超过91.6%.     

无约束手写体汉字切分方法综述

汉字切分是手写体汉字识别预处理中的一个难点，切分的正确与否直接关系到汉字识别的准确度。该义对目前无约束手写体汉字识别在行字切分上主要采用的方法作了具体介绍，闸述了各种方法的优缺点，最后根据目前的研究状况，指出今后研究中心需要注意的问题和研究的发展方向。     

基于可变窗口的古代手写汉字多步切分方法

根据古文献和古代手写体汉字纵向书写,上下字多粘连、多重叠等特点,提出一种基于可变窗口的古代手写汉字多步切分方法。对古文献进行列切分,列切分采用投影过滤的方法,采用投影法对手写字进行粗切分,对存在粘连或重叠的字串采用可变窗口的方法进行细切分,确定字串中每一个单字的切分路径。实验结果表明,该方法能够有效应用于古文献中手写汉字的切分。     

汉语语料的自动分类

语料库语言学的发展要求语料库的规模越来越大。随着电子出版业的迅速发展, 获取大量机读文本建立大规模语料库已成为可能。但是收集来的粗语料是杂乱无章的, 在作加工整理前必须分类。若用手工分类则工作量很大。本文介绍了一种语料自动分类办法。它采用文中提出的语料相关系数的概念, 并利用不同类语料相关系数不同的特点进行分类, 取得了93%的大类分类正确率。     

基于线索树双层聚类的微博话题检测

微博作为一种全新的信息发布模式,在极大程度上增强了网络信息的开放性和互动性,但同时也造成微博空间内信息量的裂变式增长。利用话题检测技术将微博文本信息按照话题进行归类和组织,可以帮助用户在动态变化的信息环境下高效获取个性信息或热点话题。该文针对微博文本短、半结构、上下文信息丰富等特点,提出了基于线索树的双层聚类的话题检测方法,通过利用融合了时序特征和作者信息的话题模型(Temporal-Author-Topic, TAT)进行线索树内的局部聚类,借以实现垃圾微博的过滤,最后利用整合后的线索树进行全局话题检测。实验结果显示该方法在解决数据稀疏方面取得了较好的效果,话题检测的F值达到31.2%。     

机器学习在汉语关联词语识别中的应用

关联词语在一些汉语议论文章中占很大的比重,因而,对于此类汉语文章的分析,关联词可以起到非常重要的作用。本文主要讨论如何将机器学习应用于汉语关联词的歧义辨别——原因,方法和效果。我们在已经加工完毕的80篇汉语语料的基础上,抽取了用于机器学习的训练集和测试集,并使用C4.5进行了测试,识别正确率在80%以上。在文章的后面,我们还从语言学的角度对机器学习的结果进行了解释和分析。     

Automated reference resolution in legal texts

This paper investigates the task of reference resolution in the legal domain. This is a new interesting task in Legal Engineering research. The goal is to create a system which can automatically detect references and then extracts their referents. Previous work limits itself to detect and resolve references at the document targets. In this paper, we go a step further in trying to resolve references to sub-document targets. Referents extracted are the smallest fragments of texts in documents, rather than the entire documents that contain the referenced texts. Based on analyzing the characteristics of reference phenomena in legal texts, we propose a four-step framework to deal with the task: mention detection, contextual information extraction, antecedent candidate extraction, and antecedent determination. We also show how machine learning methods can be exploited in each step. The final system achieves 80.06 % in the F1 score for detecting references, 85.61 % accuracy for resolving them, and 67.02 % in the F1 score for the end-to-end setting task on the Japanese National Pension Law corpus.     

利用平行网页建立中英文统计翻译模型

建立翻译模型的目的是试图从平行文本(或翻译例句)中自动抽取翻译关系。本文将描述我们在建立中英文统计翻译模型上的尝试。我们所用的平行文本是从万维网上自动获得的半结构性平行文本。在训练过程中,我们尽量利用文本中的HTML结构信息。实验表明,所训练的翻译模型能达到80%的准确率。对于象跨语言信息检索这样的应用,这样的准确率已经能大致满足需要。这一工作表明,对于检索引擎上的问句的翻译可以使用比机器翻译成本更低的工具。     

Short text topic modeling by exploring original documents

Topic modeling for short texts faces a tough challenge, owing to the sparsity problem. An effective solution is to aggregate short texts into long pseudo-documents before training a standard topic model. The main concern of this solution is the way of aggregating short texts. A recent developed self-aggregation-based topic model (SATM) can adaptively aggregate short texts without using heuristic information. However, the model definition of SATM is a bit rigid, and more importantly, it tends to overfitting and time-consuming for large-scale corpora. To improve SATM, we propose a generalized topic model for short texts, namely latent topic model (LTM). In LTM, we assume that the observable short texts are snippets of normal long texts (namely original documents) generated by a given standard topic model, but their original document memberships are unknown. With Gibbs sampling, LTM drives an adaptive aggregation process of short texts, and simultaneously estimates other latent variables of interest. Additionally, we propose a mini-batch scheme for fast inference. Experimental results indicate that LTM is competitive with the state-of-the-art baseline models on short text topic modeling.     

融合文本概念化与网络表示的观点检索

观点检索是自然语言处理领域中的一个热点研究课题.现有的观点检索模型在检索过程中往往无法根据上下文将词汇进行知识、概念层面的抽象,在语义层面忽略词汇之间的语义联系,观点层面缺乏观点泛化能力.因此,提出一种融合文本概念化与网络表示的观点检索方法.该方法首先利用知识图谱分别将用户查询和文本概念化到正确的概念空间,并利用网络表示将知识图谱中的词汇节点表示成低维向量,然后根据词向量推出查询和文本的向量并用余弦公式计算用户查询与文本的相关度,接着引入基于统计机器学习的分类方法挖掘文本的观点.最后利用概念空间、网络表示空间以及观点分析结果构建特征,并服务于观点检索模型,相关实验表明,本文提出的检索模型可以有效提高多种检索模型的观点检索性能.其中,基于统一相关模型的观点检索方法在两个实验数据集上相比基准方法在MAP评价指标上分别提升了6.1%和9.3%,基于排序学习的观点检索方法在两个实验数据集上相比于基准方法在MAP评价指标上分别提升了2.3%和14.6%.     

Emphasizing Essential Words for Sentiment Classification Based on Recurrent Neural Networks

With the explosion of online communication and publication, texts become obtainable via forums, chat messages, blogs, book reviews and movie reviews. Usually, these texts are much short and noisy without sufficient statistical signals and enough information for a good semantic analysis. Traditional natural language processing methods such as Bow-of-Word (BOW) based probabilistic latent semantic models fail to achieve high performance due to the short text environment. Recent researches have focused on the correlations between words, i.e., term dependencies, which could be helpful for mining latent semantics hidden in short texts and help people to understand them. Long short-term memory (LSTM) network can capture term dependencies and is able to remember the information for long periods of time. LSTM has been widely used and has obtained promising results in variants of problems of understanding latent semantics of texts. At the same time, by analyzing the texts, we find that a number of keywords contribute greatly to the semantics of the texts. In this paper, we establish a keyword vocabulary and propose an LSTM-based model that is sensitive to the words in the vocabulary; hence, the keywords leverage the semantics of the full document. The proposed model is evaluated in a short-text sentiment analysis task on two datasets: IMDB and SemEval-2016, respectively. Experimental results demonstrate that our model outperforms the baseline LSTM by 1%~2% in terms of accuracy and is effective with significant performance enhancement over several non-recurrent neural network latent semantic models (especially in dealing with short texts). We also incorporate the idea into a variant of LSTM named the gated recurrent unit (GRU) model and achieve good performance, which proves that our method is general enough to improve different deep learning models.     

Steganalysis against substitution-based linguistic steganography based on context clusters

Linguistic steganalysis has been an increasing interest stimulated by the emerging research area of linguistic steganography during the past few years. However, due to limitations of computer natural language processing capability, linguistic steganalysis is a challenging task. Existing steganalysis methods are inefficient to analyze most substitution-based linguistic steganography methods which preserve the syntactic and semantic correctness of cover texts. This paper provides a new steganalysis scheme against substitution-based linguistic steganography based on context clusters. In this scheme, we introduce context clusters to estimate the context fitness and show how to use the statistics of context fitness values to distinguish between normal texts and stego texts. Finally, under this scheme, we present the steganalysis method for synonym substitution-based linguistic steganography. Our experimental results show that the proposed steganalysis method can analyze synonym substitution-based linguistic steganography efficiently and the steganalysis accuracy reaches as high as 98.86%.