印刷维吾尔文本切割

我国新疆地区使用的维吾尔文借用阿拉伯文字母书写。因为阿拉伯文字母自身书写的特点,造成维文文本的切割和识别极其困难。本文在连通体分类的基础上,结合水平投影和连通体分析的方法实现维文文本的文字行切分和单词切分。然后定位单词基线位置,计算单词轮廓和基线的距离,寻找所有可能的切点实现维文单词过切割,最后利用规则合并过切分字符。实验结果表明,字符切割准确率达到99 %以上。     

一种基于HMM和统计语言模型的维吾尔文及阿拉伯文识别方法

维吾尔文和阿拉伯文是采用阿拉伯文字母的从右向左书写的连写文字。它们识别方法的研究对于多文种文本图像内容的利用具有重要意义。利用HTK工具包,分别建立基于隐马尔科夫模型HMM(Hidden Markov Model)的印刷体维吾尔文和阿拉伯文识别系统,其中特征提取部分采用分布密度特征和局部方向特征。研究利用HTK工具建立维吾尔文和阿拉伯文统计语言模型,并将语言模型用于改进识别系统性能。实验结果表明采用统计语言模型可有效提高文字识别系统性能。其中,在包含24 000个单词的印刷体维吾尔文测试集上,通过利用语言模型识别率从78.28%提高到97.45%;在包含759个单词的印刷体阿拉伯文测试集上,通过利用语言模型识别率从79.07%提高到85.80%。     

基于统计专用字符的维、哈、柯文文种识别研究

在Unicode编码方案中维、哈、柯文字符安排在阿拉伯字符区域,三种语言中共享字符比较多,跟阿拉伯字符区域混在一起,没有专用的语言ID。在信息检索和自然语言处理领域对维、哈、柯文的识别、处理带来不便。该文首先分析并总结了维、哈、柯文三种语言中的专用字符、复合字符、某些字符在某种语言中出现形势的独特性等特征,然后在此基础上设计了维、哈、柯文种识别算法。 实验结果表明该文提出的文种识别算法的正确率在文本多于70词时达到96.67%以上。     

维吾尔文信息处理平台Open Type字体制作技术

维吾尔文信息在计算机上的处理方法与维吾尔文处理系统采用的维吾尔文字符编码方案和维吾尔文字体文件的结构有非常密切的关系。在保留字体文件原有的所有语言支持功能的前提下制作符合国际UNICODE编码标准的字体文件是在Windows系列操作系流下开发维吾尔文信息处理平台的一个重要方面。讨论Windows上维吾尔文字符显示与维吾尔文字体文件结构之间的关系,深入分析Open Type字体与文本垂直显示和高级印刷功能有关的描述表,最后提出符合国际UNICODE编码标准的维吾尔文Open Type字体的制作方案。     

基于Matlab GUI的维吾尔文字符识别系统的设计

基于建立对视频中维吾尔文字符的识别提取系统,以视频中维吾尔字符为对象,首先对维吾尔文字符进行了字符检测,以便确定其维吾尔文字符在视频中的大小、位置,其次对所确定的维吾尔文字符进行字符定位以便作进一步的处理,然后运用Canny算子将其提取出的维吾尔文字符进行彩色图像转换为灰度化的边缘图像,最后利用Matlab GUI设计了维吾尔文字符的识别系统,实现了视频中维吾尔文字符的检测、字符定位、图像抖动、对比度调整、灰度化、边缘检测、提取字符等功能.实验结果均显示出该算法的优良性能,并证明了该识别系统性能的稳定性和极其良好的扩展性,为维吾尔文字符的识别研究提供了一个简单有效的仿真平台.     

维、哈、柯多文种考试报名信息采集系统设计

论文叙述了基于阿拉伯字母的维吾尔文、哈萨克文、柯尔克孜文的OMR信息采集系统的设计方法。首次提出了考试报名信息采集系统中采用的光标识别维吾尔文编码方案,以及开发工具PB10.0版本基于Unicode编码的维、哈、克文信息在数据库信息处理方面要求,光标识别维吾尔文编码的转换处理方法。论文提出的编码方案和转换方法,对基于阿拉伯字母的维、哈、柯文信息采集应用领域有一定的指导和参考意义。     

基于多策略的维吾尔文网页识别方法

经过对大量维吾尔文网站的调查与分析,该文从多语种混合网页中针对维吾尔文网页识别进行了研究,这对维吾尔语信息处理工作起着关键作用。首先该文探讨了维吾尔文不规范网页的字符编码转换规则及原理,以此对不规范维吾尔文字符进行了相应的处理,之后介绍了基于修改的N-Gram方法和基于维吾尔语常用词特征向量的两种方法,其中后者融合了维吾尔文常用候选词语料库及向量空间模型(Vector Space Model)。使用三种不同类型的维吾尔文网页文本作为本研究的数据集,在此基础上验证了该文提出的网页识别方法,以及采用不同的方法进行了网页识别的实验。实验结果表明,基于N-Gram的方法对正文较长的新闻或论坛网页的识别性能最佳,反而基于常用词特征向量的方法对短文本的网页识别性能优越N-Gram。所提方法对维吾尔文网页识别的整体性能达到90%以上,并验证了这两种方法的有效性。     

基于连通域特征的维吾尔手写文本行分割

针对维吾尔文手写体文本中行分割问题,基于连通域大小将图像中文字分为三类,提出了自适应涂抹细化算法,对主体文本行进行定位;并对第三类连通域中相邻两文本行间粘连的字符进行切割;此外,利用重心范围内的邻域搜索算法,解决了剩余笔画的文本行归附问题。实验结果表明,该方法与常见的水平投影法,分段投影法,及涂抹方法相比具有更好的分割效果。     

多字体印刷维吾尔文的切分

在许多文字识别系统中, 字符切分是预处理阶段的一部分, 其目的是从文本图象中分离出字母图象。而后才能针对切分后的每个字母进行识别。在具有连体特征的文字中, 字符切分就显得特别重要, 因为字符切分的准确与否直接影响字符的识别。维吾尔文就具有这种明显的连体特点, 本文主要讨论了采用抽取投影特征的方法, 实现了多字体维吾尔文的行切分、字切分和字符切分。     

基于统计机器翻译的汉维词对齐研究

描述了一个基于统计机器翻译的汉维词对齐系统。系统处理过程分为两个模块:预处理和词对齐。预处理过程包括汉文文本预处理和维吾尔文文本预处理,其中维吾尔文文本预处理过程为:首先将维吾尔文转换成拉丁维文,然后将拉丁维文中个别字符替换为无歧义的字符。词对齐实现过程:首先利用IBM Model1-3,然后结合Och等人提出的启发式的思路进行优化,构建基于统计机器翻译的汉维词对齐系统。实验结果表明此系统可行。     

维哈柯及蒙语多文种语言相似性考查研究

该文以阿勒泰语系下的维哈柯及蒙古语多语言平行文本和语音语料为研究对象,分别对比多语言文本量化序列向量及语音声学音律特征的相似度,研究语言信息间存在的相通性。试验发现,同语系同语族黏着语言相似度较高 文本相似性达85%;声频特征相似性达95%。从而确认在同语系多种黏着语言间创建语言信息共享云模的可行性,这将有利于实现语言文本及语音信息的跨语言转换处理,极大降低少数民族语言信息处理成本。     

基于稳健词素序列和LSTM的维吾尔语短文本分类

维吾尔语是一种派生类语言,其词是由词干和词缀连接而成的。其中,词干是有实际意义的词汇单元,词缀提供语法功能。该文提出了基于词干单元和长短期记忆(LSTM)网络的维吾尔语短文本分类技术。用基于词-词素平行训练语料的稳健词素切分和词干提取方法,从互联网下载的文本中提取其词干,以此构建词干序列文本语料库,并通过Word2Vec算法映射到实数向量空间。然后用LSTM网络作为特征选择和文本分类算法进行维吾尔语短文本分类实验,并得到95.48%的分类准确率。从实验结果看,对于维吾尔语等派生类语言而言,特别是对于带噪声的文本,基于词干的分类方法有更多优异的性能。     

基于深度学习的相似语言短文本的语种识别方法

在语种识别中,传统的N-Gram方法对文本长度依赖度高,因而无法有效地对短文本进行语种识别。现有的基于神经网络的模型无法同时考虑词本身信息和词间组合信息,从而降低了短文本语种识别的质量。针对以上问题,提出一种基于深度学习的字符级短文本语种识别方法。采用卷积神经网络从字符向量中获取词中字符组合信息;通过长短期记忆网络获取词与词之间的特征信息;使用全连接网络实现相似语言的语种识别。在维吾尔语、哈萨克语以及DSL2017数据集上的实验结果表明,该方法可以有效地提高相似语言短文本的识别精度。     

维吾尔语形态分析研究综述

维吾尔语具有形态丰富性、黏着性和数据稀疏性等特点,处理技术和英汉等热门语言有着较大差距并且未能满足新疆地区发展需求。形态分析是自然语言处理的重要组成部分,研究维吾尔语形态分析对于推动维吾尔语信息处理技术发展有着重要意义。简述了维吾尔语语法,描述了维吾尔语自然语言处理、形态分析及其相关基本资源研究现状,将常见方法分为基于规则、基于词典、基于统计、基于深度学习和基于混合5大类并分析了各种方法的优劣,介绍了维吾尔语形态分析后续研究,借鉴了先进的词法分析方法,总结了维吾尔语形态分析面临的挑战和机遇,并对其未来发展趋势进行展望。     

Towards an open platform based on HPSG formalism for the standard Arabic language

The aim of this paper is to present an open software platform for analysing texts in standard Arabic language. The originality of this platform is that it is an integrated software environment which offers all the necessary resources and tools for parsing Arabic texts. For formalising the several elements of the language, the HPSG formalism has been adopted because of its effectiveness and its ability to be adapted to any natural language. Currently, the platform is operational with an appreciable coverage of many Arabic syntactic structures. In the medium-term, our objective is to use the platform for developing applications for the Arabic language such as interfaces, learning, information retrieval…etc.     

Sentence alignment using P-NNT and GMM

Parallel corpora have become an essential resource for work in multilingual natural language processing. However, sentence aligned parallel corpora are more efficient than non-aligned parallel corpora for cross-language information retrieval and machine translation applications. In this paper, we present two new approaches to align English–Arabic sentences in bilingual parallel corpora based on probabilistic neural network (P-NNT) and Gaussian mixture model (GMM) classifiers. A feature vector is extracted from the text pair under consideration. This vector contains text features such as length, punctuation score, and cognate score values. A set of manually prepared training data was assigned to train the probabilistic neural network and Gaussian mixture model. Another set of data was used for testing. Using the probabilistic neural network and Gaussian mixture model approaches, we could achieve error reduction of 27% and 50%, respectively, over the length based approach when applied on a set of parallel English–Arabic documents. In addition, the results of (P-NNT) and (GMM) outperform the results of the combined model which exploits length, punctuation and cognates in a dynamic framework. The GMM approach outperforms Melamed and Moore’s approaches too. Moreover these new approaches are valid for any languages pair and are quite flexible since the feature vector may contain more, less or different features, such as a lexical matching feature and Hanzi characters in Japanese–Chinese texts, than the ones used in the current research.     

Region growing based segmentation algorithm for typewritten and handwritten text recognition

This paper presents a new technique of high accuracy to recognize both typewritten and handwritten English and Arabic texts without thinning. After segmenting the text into lines (horizontal segmentation) and the lines into words, it separates the word into its letters. Separating a text line (row) into words and a word into letters is performed by using the region growing technique (implicit segmentation) on the basis of three essential lines in a text row. This saves time as there is no need to skeletonize or to physically isolate letters from the tested word whilst the input data involves only the basic information—the scanned text. The baseline is detected, the word contour is defined and the word is implicitly segmented into its letters according to a novel algorithm described in the paper. The extracted letter with its dots is used as one unit in the system of recognition. It is resized into a 9 × 9 matrix following bilinear interpolation after applying a lowpass filter to reduce aliasing. Then the elements are scaled to the interval [0,1]. The resulting array is considered as the input to the designed neural network. For typewritten texts, three types of Arabic letter fonts are used—Arial, Arabic Transparent and Simplified Arabic. The results showed an average recognition success rate of 93% for Arabic typewriting. This segmentation approach has also found its application in handwritten text where words are classified with a relatively high recognition rate for both Arabic and English languages. The experiments were performed in MATLAB and have shown promising results that can be a good base for further analysis and considerations of Arabic and other cursive language text recognition as well as English handwritten texts. For English handwritten classification, a success rate of about 80% in average was achieved while for Arabic handwritten text, the algorithm performance was successful in about 90%. The recent results have shown increasing success for both Arabic and English texts.     

现代维吾尔语常用词统计关键技术研究

本文研究了构建现代维吾尔语语料库的关键技术与方法,特别是现代维吾尔语语料库的构建,并对现代维吾尔语语料预处理技术,现代维吾尔语语料统计技术,现代维吾尔语词干提取技术,现代维吾尔语数据分析技术进行了研究;研制了现代维吾尔语常用词候选表, 从词语的使用频度和词语的分布两方面对词语进行了基本考察,将维吾尔语词语的“词种数、频次、频率、文本数、词长”作为常用词候选表的依据。     

Exploring and exploiting a historical corpus for Arabic

This paper presents a historical Arabic corpus named HAC. At this early embryonic stage of the project, we report about the design, the architecture and some of the experiments which we have conducted on HAC. The corpus, and accordingly the search results, will be represented using a primary XML exchange format. This will serve as an intermediate exchange tool within the project and will allow the user to process the results offline using some external tools. HAC is made up of Classical Arabic texts that cover 1600 years of language use; the Quranic text, Modern Standard Arabic texts, as well as a variety of monolingual Arabic dictionaries. The development of this historical corpus assists linguists and Arabic language learners to effectively explore, understand, and discover interesting knowledge hidden in millions of instances of language use. We used techniques from the field of natural language processing to process the data and a graph-based representation for the corpus. We provided researchers with an export facility to render further linguistic analysis possible.