Semi-automatic documentation of an implemented linguistic grammar augmented with a treebank

We have constructed a large scale and detailed database of lexical types in Japanese from a treebank that includes detailed linguistic information. The database helps treebank annotators and grammar developers to share precise knowledge about the grammatical status of words that constitute the treebank, allowing for consistent large-scale treebanking and grammar development. In addition, it clarifies what lexical types are needed for precise Japanese NLP on the basis of the treebank. In this paper, we report on the motivation and methodology of the database construction.

Building the essential resources for Finnish: the Turku Dependency Treebank

In this paper, we present the final version of a publicly available treebank of Finnish, the Turku Dependency Treebank. The treebank contains 204,399 tokens (15,126 sentences) from 10 different text sources and has been manually annotated in a Finnish-specific version of the well-known Stanford Dependency scheme. The morphological analyses of the treebank have been assigned using a novel machine learning method to disambiguate readings given by an existing tool. As the second main contribution, we present the first open source Finnish dependency parser, trained on the newly introduced treebank. The parser achieves a labeled attachment score of 81 %. The treebank data as well as the parsing pipeline are available under an open license at http://bionlp.utu.fi/.     

基于“词—词性”匹配模式获取的古汉语树库快速构建方法

该文针对古汉语文本小、句简短、模式性强的结构特点,提出了一种基于“词-词性”匹配模式获取的快速树库构建方法,将句法标注过程规约为获取候选匹配模式、制定句法转换规则、自动生成句法树和最终人工校对四个步骤。该方法可大大缩减人工标注工作量,节省树库构建的工程成本,且所获取的匹配规则在古汉语教学研究中具有一定的实用价值。     

Treebanks gone bad

This article describes how a treebank of ungrammatical sentences can be created from a treebank of well-formed sentences. The treebank creation procedure involves the automatic introduction of frequently occurring grammatical errors into the sentences in an existing treebank, and the minimal transformation of the original analyses in the treebank so that they describe the newly created ill-formed sentences. Such a treebank can be used to test how well a parser is able to ignore grammatical errors in texts (as people do), and can be used to induce a grammar capable of analysing such sentences. This article demonstrates these two applications using the Penn Treebank. In a robustness evaluation experiment, two state-of-the-art statistical parsers are evaluated on an ungrammatical version of Sect. 23 of the Wall Street Journal (WSJ) portion of the Penn treebank. This experiment shows that the performance of both parsers degrades with grammatical noise. A breakdown by error type is provided for both parsers. A second experiment retrains both parsers using an ungrammatical version of WSJ Sections 2–21. This experiment indicates that an ungrammatical treebank is a useful resource in improving parser robustness to grammatical errors, but that the correct combination of grammatical and ungrammatical training data has yet to be determined.     

面向机器翻译的藏语短语句法研究

针对现代藏语句法,在参照宾大中文树库的基础上,构建藏语短语句法树库,并建立了树库编辑工具,为藏汉机器翻译服务。在短语句法树库的基础上,提出一种融合藏语句法特征的藏汉机器翻译方法。实验分析结果表明,该方法可以很好地应用于藏汉机器翻译系统。     

Hindi CCGbank: A CCG treebank from the Hindi dependency treebank

In this paper, we present an approach for automatically creating a combinatory categorial grammar (CCG) treebank from a dependency treebank for the subject–object–verb language Hindi. Rather than a direct conversion from dependency trees to CCG trees, we propose a two stage approach: a language independent generic algorithm first extracts a CCG lexicon from the dependency treebank. An exhaustive CCG parser then creates a treebank of CCG derivations. We also discuss special cases of this generic algorithm to handle linguistic phenomena specific to Hindi. In doing so we extract different constructions with long-range dependencies like coordinate constructions and non-projective dependencies resulting from constructions like relative clauses, noun elaboration and verbal modifiers.     

基于HMM的汉语介词短语自动识别研究

提出了一种基于隐马尔可夫模型(HMM)的介词短语界定模型,通过HMM的介词短语边界自动识别和依存语法错误校正2个处理阶段,较好地完成了对一个经过分词和词性标注的句子进行介词短语界定任务,为更进一步的句法分析工作打下良好的基础。试验结果显示：该模型的识别正确率达到了86.5%（封闭测试）和77.7%(开放测试),取得了令人满意的结果。     

The Spanish DELPH-IN grammar

In this article we present a Spanish grammar implemented in the Linguistic Knowledge Builder system and grounded in the theoretical framework of Head-driven Phrase Structure Grammar. The grammar is being developed in an international multilingual context, the DELPH-IN Initiative, contributing to an open-source repository of software and linguistic resources for various Natural Language Processing applications. We will show how we have refined and extended a core grammar, derived from the LinGO Grammar Matrix, to achieve a broad-coverage grammar. The Spanish DELPH-IN grammar is the most comprehensive grammar for Spanish deep processing, and it is being deployed in the construction of a treebank for Spanish of 60,000 sentences based in a technical corpus in the framework of the European project METANET4U (Enhancing the European Linguistic Infrastructure, GA 270893GA; http://www.meta-net.eu/projects/METANET4U/.) and a smaller treebank of about 15,000 sentences based in a corpus from the press.     

Ensuring annotation consistency and accuracy for Vietnamese treebank

Treebanks are important resources for researchers in natural language processing. They provide training and testing materials so that different algorithms can be compared. However, it is not a trivial task to construct high-quality treebanks. We have not yet had a proper treebank for such a low-resource language as Vietnamese, which has probably lowered the performance of Vietnamese language processing. We have been building a consistent and accurate Vietnamese treebank to alleviate such situations. Our treebank is annotated with three layers: word segmentation, part-of-speech tagging, and bracketing. We developed detailed annotation guidelines for each layer by presenting Vietnamese linguistic issues as well as methods of addressing them. Here, we also describe approaches to controlling annotation quality while ensuring a reasonable annotation speed. We specifically designed an appropriate annotation process and an effective process to train annotators. In addition, we implemented several support tools to improve annotation speed and to control the consistency of the treebank. The results from experiments revealed that both inter-annotator agreement and accuracy were higher than 90%, which indicated that the treebank is reliable.     

基于树库转换的藏语依存句法树库构建方法

构建藏语依存树库是实现藏语句法分析的重要基础,对藏语本体研究和信息处理具有重要价值。基于此,该文提出了一种基于树库转换的藏语依存树库构建方法。该方法首先扩充了前期构建的藏语短语结构树库,然后根据藏语短语结构树和依存树的特征设计树库转换规则,实现藏语短语结构树到依存结构树的初步转换,最后对自动转换结果进行人工校验,得到了2.2万句藏语依存树。为了对转换结果做出量化评价,该文抽取了依存树库中5%的依存树,对其依存关系进行校验和统计,最终依存关系的准确率达到89.36%,中心词的准确率达到92.09%。此外,该文使用基于神经网络的句法分析模型验证了依存树库的有效性。在该模型上,UAS值和LAS值分别达到83.62%和81.90%。研究证明,使用半自动的树库转换方法能够有效地完成藏语依存树库构建工作。     

短语结构树库向依存结构树库转化研究

汉语依存树库的建设相对其他语言如英语,在规模和质量上还有一些差距。树库标注需要付出很大的人力物力,并且保证树库质量也比较困难。该文尝试通过规则和统计相结合的方法,将宾州汉语短语树库Penn Chinese Treebank转化为哈工大依存树库HIT-IR-CDT的体系结构,从而增大现有依存树库的规模。将转化后的树库加入HIT-IR-CDT,训练和测试依存句法分析器的性能。实验表明,加入少量经转化后的树库后,依存句法分析器的性能有所提高;但加入大量树库后,性能反而下降。经过细致分析,作为一种利用多种树库提高依存句法分析器性能的方法,短语转依存还存在很多需要深入研究的方面。     

短语树到依存树的自动转换研究

不同标注体系的树库之间的相互转换是计算语言学研究的重要内容之一。本文在总结国内外几种树库标注体系及相互转换实践的基础上,结合清华汉语树库(Tsinghua Chinese Treebank ,简称TCT) 标注体系的特点,提出了一种将TCT从短语结构转换成依存结构(Dependency Structure) 的算法。这种算法充分利用了TCT具有的功能、结构的双重标记,转换得到的依存树不仅包含了各个节点之间相互依存的层次关系,更包含了相互依存的两个节点的具体的依存关系类型。我们对转换的效果进行了抽样评估,准确率可以达到97137 %。     

A feature-based approach to better automatic treebank conversion

In the field of constituency parsing, there exist multiple human-labeled treebanks which are built on non-overlapping text samples and follow different annotation standards. Due to the extreme cost of annotating parse trees by human, it is desirable to automatically convert one treebank (called source treebank) to the standard of another treebank (called target treebank) which we are interested in. Conversion results can be manually corrected to obtain higher-quality annotations or can be directly used as additional training data for building syntactic parsers. To perform automatic treebank conversion, we divide constituency parses into two separate levels: the part-of-speech (POS) and syntactic structure (bracketing structures and constituent labels), and conduct conversion on these two levels respectively with a feature-based approach. The basic idea of the approach is to encode original annotations in a source treebank as guide features during the conversion process. Experiments on two Chinese treebanks show that our approach can convert POS tags and syntactic structures with the accuracy of 96.6 and 84.8 %, respectively, which are the best reported results on this task.     

Parsing Arabic using induced probabilistic context free grammar

The importance of the parsing task for NLP applications is well understood. However developing parsers remains difficult because of the complexity of the Arabic language. Most parsers are based on syntactic grammars that describe the syntactic structures of a language. The development of these grammars is laborious and time consuming. In this paper we present our method for building an Arabic parser based on an induced grammar, PCFG grammar. We first induce the PCFG grammar from an Arabic Treebank. Then, we implement the parser that assigns syntactic structure to each input sentence. The parser is tested on sentences extracted from the treebank (1650 sentences).We calculate the precision, recall and f-measure. Our experimental results showed the efficiency of the proposed parser for parsing modern standard Arabic sentences (Precision: 83.59 %, Recall: 82.98 % and F-measure: 83.23 %).     

基于依存语法构建多视图汉语树库

树库是自然语言处理中一项重要的基础资源,现有树库基本上都是单视图树,支持短语结构语法或者依存语法。该文提出一套基于依存语法的多视图汉语树库标注体系,仅需标注中心语和语法角色两类信息,之后可以自动地推导出描述句法结构所需的短语结构功能和层次信息,从而可以在不增加标注工作量的前提下获得更多语法信息。基于该体系,构建了北京大学多视图汉语树库(PMT)1.0版,含有64000句、140万词,支持短语结构语法和依存语法两个视图。     

汉语树库的构建

本文讨论了汉语树库构建的若干基础问题, 包括一个适合于自动分析和人工标注的汉语句法标记集、汉语树库加工处理规范和人机互助的树库加工模型, 介绍了一个已经实现的汉语自动句法标注系统, 和在此基础上进行的一些树库构建实验, 最后提出了构建大规模汉语树库的设想。     

多领域中文依存树库构建与影响统计句法分析因素之分析

为提升依存分析并分析影响其精度的相关因素,该文构建了大规模中文通用依存树库和中等规模领域依存树库。基于这一系列树库,通过句法分析实验考察质量、规模、领域差异等因素对中文依存分析的影响,实验结果表明: (1)树库规模和质量均与句法分析精度成正相关关系,质量应先于规模因素被优先考虑;(2)通用树库和领域树库之间的差异程度与前者对后者的替代性成相关关系;(3)两种树库混合使用的效果同样与领域差异有关。
     

Co-training机器学习方法在中文组块识别中的应用

采用半指导机器学习方法co2training 实现中文组块识别。首先明确了中文组块的定义,co-training 算法的形式化定义。文中提出了基于一致性的co-training 选取方法将增益的隐马尔可夫模型(Transductive HMM) 和基于转换规则的分类器(fnTBL) 组合成一个分类体系,并与自我训练方法进行了比较,在小规模汉语树库语料和大规模未带标汉语语料上进行中文组块识别,实验结果要比单纯使用小规模的树库语料有所提高,F 值分别达到了85134 %和83141 % ,分别提高了2113 %和7121 %。     

汉语句法树库标注体系

语料库的句法标注是语料库语言学研究的前沿课题。本文在研究和总结国内外句法树库标注实践的基础上,提出了一套汉语真实文本的句法树标注体系。它以完整的层次结构树为基础,对句法树上的每个非终结符节点都给出两个标记:成分标记和关系标记,形成双标记集的句法信息描述体系。目前,这两个标记集分别包含了16和27个标记,对汉语句子的不同句法组合的外部功能分布和内部组合特点进行了详细描述。在此基础上,我们开发完成了100万词规模的汉语句法树库TCT,对其中各种复杂语言现象的标注实践显示了这套标注体系具有很好的信息覆盖率和语料适应性。