Quality estimation for machine translation: some lessons learned

The dissemination of statistical machine translation (SMT) systems in the professional translation industry is still limited by the lack of reliability of SMT outputs, the quality of which varies to a great extent. A critical piece of information would be for MT systems to automatically assess their output translations with automatically derived quality measures. Predicting quality measures was indeed the goal of a shared task at the Workshop on SMT in 2012. In this contribution, we first report our results for this shared task, detailing the features that we found to be the most predictive of quality. In the latter part, we reexamine the shared task data and protocol and show that several factors actually contributed to the difficulty of the task, and discuss alternative evaluation designs.     

PHRASE AND NGRAM-BASED STATISTICAL MACHINE TRANSLATION SYSTEM COMBINATION

Multiples translations can be computed by one machine translation (MT) system or by different MT systems. We may assume that different MT systems make different errors due to using different models, generation strategies, or tweaks. An investigated technique, inherited from automatic speech recognition (ASR), is the so-called system combination that is based on combining the outputs of multiples MT systems. We combine the outputs of a phrase- and Ngram-based Statistical MT (SMT) systems using statistical criteria and additional rescoring features.     

Hybrid Architectures for Machine Translation Systems

Although some progress has been made on the quality of Machine Translation in recent years, there is still a significant potential for quality improvement. There has also been a shift in paradigm of machine translation, from “classical” rule-based systems like METAL or LMT¹ towards example-based or statistical MT.² It seems to be time now to evaluate the progress and compare the results of these efforts, and draw conclusions for further improvements of MT quality.The paper starts with a comparison between statistical MT (henceforth: SMT) and rule-based MT (henceforth: RMT) systems, and describes the set-up and the evaluation results; the second section analyses the strengths and weaknesses of the respective approaches, and the third one discusses models of an architecture for a hybrid system.     

Assembling translations from multi-engine machine translation outputs

In this paper, we present a hybrid architecture for developing a system combination model that works in three layers to achieve better translated outputs. In the first layer, we have various machine translation models (i.e. Neural Machine Translation (NMT), Statistical Machine Translation (SMT), etc.). In the second layer, the outputs of these models are combined to leverage the advantages of both the systems (i.e SMT and NMT systems) by using the statistical approach and neural-based approach. But each approach has some advantages and limitations. So, instead of selecting an individual combined system’s output as the final one, we apply these outputs in the final layer to produce the target output by assigning appropriate preferences to SMT based and neural-based combinations. Though there are some techniques for system combination but no such approach exists which uses preferences from various system combination models (statistical and neural) for the purpose of better assembling. Empirical results show improved performance in the terms of translation accuracy. Our experiments on two benchmark datasets of English–Hindi and Hindi–English pairs show that the proposed model performs significantly better than the participating models. Apparently, the efficacy of proposed model is significantly better than the state-of-the art machine translation combination systems (6.10 and 4.69 BLEU points for English-to-Hindi, and Hindi-to-English, respectively).     

Factor-based evaluation for English to Hindi MT outputs

Design and implementation of automatic evaluation methods is an integral part of any scientific research in accelerating the development cycle of the output. This is no less true for automatic machine translation (MT) systems. However, no such global and systematic scheme exists for evaluation of performance of an MT system. The existing evaluation metrics, such as BLEU, METEOR, TER, although used extensively in literature have faced a lot of criticism from users. Moreover, performance of these metrics often varies with the pair of languages under consideration. The above observation is no less pertinent with respect to translations involving languages of the Indian subcontinent. This study aims at developing an evaluation metric for English to Hindi MT outputs. As a part of this process, a set of probable errors have been identified manually as well as automatically. Linear regression has been used for computing weight/penalty for each error, while taking human evaluations into consideration. A sentence score is computed as the weighted sum of the errors. A set of 126 models has been built using different single classifiers and ensemble of classifiers in order to find the most suitable model for allocating appropriate weight/penalty for each error. The outputs of the models have been compared with the state-of-the-art evaluation metrics. The models developed for manually identified errors correlate well with manual evaluation scores, whereas the models for the automatically identified errors have low correlation with the manual scores. This indicates the need for further improvement and development of sophisticated linguistic tools for automatic identification and extraction of errors. Although many automatic machine translation tools are being developed for many different language pairs, there is no such generalized scheme that would lead to designing meaningful metrics for their evaluation. The proposed scheme should help in developing such metrics for different language pairs in the coming days.     

藏汉神经网络机器翻译研究

神经网络机器翻译是最近几年提出的机器翻译方法,在多数语言对上逐渐超过了统计机器翻译方法,成为当前机器翻译研究前沿热点。该文在藏汉语对上进行了基于注意力的神经网络机器翻译的实验,并采用迁移学习方法缓解藏汉平行语料数量不足问题。实验结果显示,该文提出的迁移学习方法简单有效,相比短语统计机器翻译方法,提高了三个BLEU值。从译文分析中可以看出藏汉神经网络机器翻译的译文比较流畅,远距离调序能力较强,同时也存在过度翻译、翻译不充分、翻译忠实度较低等神经网络机器翻译的共同不足之处。     

Incorporating target language semantic roles into a string-to-tree translation model

The string-to-tree model is one of the most successful syntax-based statistical machine translation (SMT) models. It models the grammaticality of the output via target-side syntax. However, it does not use any semantic information and tends to produce translations containing semantic role confusions and error chunk sequences. In this paper, we propose two methods to use semantic roles to improve the performance of the string-to-tree translation model: (1) adding role labels in the syntax tree; (2) constructing a semantic role tree, and then incorporating the syntax information into it. We then perform string-to-tree machine translation using the newly generated trees. Our methods enable the system to train and choose better translation rules using semantic information. Our experiments showed significant improvements over the state-of-the-art string-to-tree translation system on both spoken and news corpora, and the two proposed methods surpass the phrase-based system on large-scale training data.     

Measuring machine translation quality as semantic equivalence: A metric based on entailment features

Current evaluation metrics for machine translation have increasing difficulty in distinguishing good from merely fair translations. We believe the main problem to be their inability to properly capture meaning: A good translation candidate means the same thing as the reference translation, regardless of formulation. We propose a metric that assesses the quality of MT output through its semantic equivalence to the reference translation, based on a rich set of match and mismatch features motivated by textual entailment. We first evaluate this metric in an evaluation setting against a combination metric of four state-of-the-art scores. Our metric predicts human judgments better than the combination metric. Combining the entailment and traditional features yields further improvements. Then, we demonstrate that the entailment metric can also be used as learning criterion in minimum error rate training (MERT) to improve parameter estimation in MT system training. A manual evaluation of the resulting translations indicates that the new model obtains a significant improvement in translation quality.     

Controlled Translation in an Example-based Environment: What do Automatic Evaluation Metrics Tell Us?

This paper presents an extended, harmonised account of our previous work on integrating controlled language data in an Example-based Machine Translation system. Gough and Way in MT Summit pp. 133–140 (2003) focused on controlling the output text in a novel manner, while Gough and Way (9th Workshop of the EAMT, (2004a), pp. 73–81) sought to constrain the input strings according to controlled language specifications. Our original sub-sentential alignment algorithm could deal only with 1:1 matches, but subsequent refinements enabled n:m alignments to be captured. A direct consequence was that we were able to populate the system’s databases with more than six times as many potentially useful fragments. Together with two simple novel improvements – correcting a small number of mistranslations in the lexicon, and allowing multiple translations in the lexicon – translation quality improves considerably. We provide detailed automatic and human evaluations of a number of experiments carried out to test the quality of the system. We observe that our system outperforms the rule-based on-line system Logomedia on a range of automatic evaluation metrics, and that the ‘best’ translation candidate is consistently highly ranked by our system. Finally, we note in a number of tests that the BLEU metric gives objectively different results than other automatic evaluation metrics and a manual evaluation. Despite these conflicting results, we observe a preference for controlling the source data rather than the target translations.     

Parallel sentence generation from comparable corpora for improved SMT

A parallel corpus is an essential resource for statistical machine translation (SMT) but is often not available in the required amounts for all domains and languages. An approach is presented here which aims at producing parallel corpora from available comparable corpora. An SMT system is used to translate the source-language part of a comparable corpus and the translations are used as queries to conduct information retrieval from the target-language side of the comparable corpus. Simple filters are then used to score the SMT output and the IR-returned sentence with the filter score defining the degree of similarity between the two. Using SMT system output gives the benefit of trying to correct one of the common errors by sentence tail removal. The approach was applied to Arabic–English and French–English systems using comparable news corpora and considerable improvements were achieved in the BLEU score. We show that our approach is independent of the quality of the SMT system used to make the queries, strengthening the claim of applicability of the approach for languages and domains with limited parallel corpora available to start with. We compare our approach with one of the earlier approaches and show that our approach is easier to implement and gives equally good improvements.     

基于句子相似度的机器翻译评价方法及其有效性分析

为了帮助开发面向本地化的基于实例的机器翻译(EBMT)系统,提出利用句子相似度评价译文质量的方法．该方法能够较有效地区分不同翻译质量的译文,从而帮助开发者发现EBMT系统存在的问题．为了证明该评价方法的有效性,对比了6个机器翻译系统人工评价和自动评价的结果．通过相关分析和显著性检验,表明了此方法和人工方法存在高度的一致性．最后提出在自动评价结果的基础上利用线性回归预测人工评分方法．     

基于句子级BLEU指标挑选数据的半监督神经机器翻译*

在单语语料的使用上,统计机器翻译可通过利用语言模型提高性能,而神经机器翻译很难通过这种方法有效利用单语语料.针对此问题,文中提出基于句子级双语评估替补(BLEU)指标挑选数据的半监督神经网络翻译模型.分别利用统计机器翻译和神经机器翻译模型对无标注数据生成候选翻译,然后通过句子级BLEU指标挑选单语候选翻译,加入到有标注的数据集中进行半监督联合训练.实验表明,文中方法能高效利用无标注的单语语料,在NIST汉英翻译任务上,相比仅使用精标的有标注数据单系统,文中方法BLEU值有所提升.     

Predicting sentence translation quality using extrinsic and language independent features

We develop a top performing model for automatic, accurate, and language independent prediction of sentence-level statistical machine translation (SMT) quality with or without looking at the translation outputs. We derive various feature functions measuring the closeness of a given test sentence to the training data and the difficulty of translating the sentence. We describe mono feature functions that are based on statistics of only one side of the parallel training corpora and duo feature functions that incorporate statistics involving both source and target sides of the training data. Overall, we describe novel, language independent, and SMT system extrinsic features for predicting the SMT performance, which also rank high during feature ranking evaluations. We experiment with different learning settings, with or without looking at the translations, which help differentiate the contribution of different feature sets. We apply partial least squares and feature subset selection, both of which improve the results and we present ranking of the top features selected for each learning setting, providing an exhaustive analysis of the extrinsic features used. We show that by just looking at the test source sentences and not using the translation outputs at all, we can achieve better performance than a baseline system using SMT model dependent features that generated the translations. Furthermore, our prediction system is able to achieve the $2$ nd best performance overall according to the official results of the quality estimation task (QET) challenge when also looking at the translation outputs. Our representation and features achieve the top performance in QET among the models using the SVR learning model.     

Symbolic-to-statistical hybridization: extending generation-heavy machine translation

The last few years have witnessed an increasing interest in hybridizing surface-based statistical approaches and rule-based symbolic approaches to machine translation (MT). Much of that work is focused on extending statistical MT systems with symbolic knowledge and components. In the brand of hybridization discussed here, we go in the opposite direction: adding statistical bilingual components to a symbolic system. Our base system is Generation-heavy machine translation (GHMT), a primarily symbolic asymmetrical approach that addresses the issue of Interlingual MT resource poverty in source-poor/target-rich language pairs by exploiting symbolic and statistical target-language resources. GHMT’s statistical components are limited to target-language models, which arguably makes it a simple form of a hybrid system. We extend the hybrid nature of GHMT by adding statistical bilingual components. We also describe the details of retargeting it to Arabic–English MT. The morphological richness of Arabic brings several challenges to the hybridization task. We conduct an extensive evaluation of multiple system variants. Our evaluation shows that this new variant of GHMT—a primarily symbolic system extended with monolingual and bilingual statistical components—has a higher degree of grammaticality than a phrase-based statistical MT system, where grammaticality is measured in terms of correct verb-argument realization and long-distance dependency translation.     

基于句对质量和覆盖度的统计机器翻译训练语料选取

该文研究的目的是在待翻译文本未知的情况下,从已有的大规模平行语料中选取一个高质量的子集作为统计机器翻译系统的训练语料,以降低训练和解码代价。该文综合覆盖度和句对翻译质量两方面因素,提出一种从已有平行语料中获取高质量小规模训练子集的方法。在CWMT2008汉英翻译任务上的实验结果表明,利用本文的方法能够从现有大规模语料中选取高质量的子集,在减少80%训练语料的情况下达到与Baseline系统(使用全部训练语料)相当的翻译性能(BLEU值)。     

System Combination for Machine Translation of Spoken and Written Language

This paper describes an approach for computing a consensus translation from the outputs of multiple machine translation (MT) systems. The consensus translation is computed by weighted majority voting on a confusion network, similarly to the well-established ROVER approach of Fiscus for combining speech recognition hypotheses. To create the confusion network, pairwise word alignments of the original MT hypotheses are learned using an enhanced statistical alignment algorithm that explicitly models word reordering. The context of a whole corpus of automatic translations rather than a single sentence is taken into account in order to achieve high alignment quality. The confusion network is rescored with a special language model, and the consensus translation is extracted as the best path. The proposed system combination approach was evaluated in the framework of the TC-STAR speech translation project. Up to six state-of-the-art statistical phrase-based translation systems from different project partners were combined in the experiments. Significant improvements in translation quality from Spanish to English and from English to Spanish in comparison with the best of the individual MT systems were achieved under official evaluation conditions.      

OpenLogos machine translation: philosophy,model, resources and customization

This paper reviews the OpenLogos rule-based machine translation system, and describes its model architecture as an incremental pipeline process. The paper also describes OpenLogos resources and their customization to specific application domains. One of the key aspects of rule-based machine translation systems intelligence is the symbology employed by these systems in representing natural language internally. The paper offers details about the OpenLogos semantico-syntactic abstract representation language known as SAL. The paper also shows how OpenLogos has addressed classic problems of rule-based machine translation, such as the cognitive complexity and ambiguity encountered in natural language processing, illustrating how SAL helps overcome them in ways distinct from other existing rule-based machine translation systems. The paper illustrates how the intelligence inherent in SAL contributes to translation quality, presenting examples of OpenLogos output of a kind that non-linguistic systems would likely have difficulty emulating. The paper shows the unique manner in which OpenLogos applies the rulebase to the input stream and the kind of results produced that are characteristic of the OpenLogos output. Finally, the paper deals with an important advantage of rule-based machine translation systems, namely, the customization and adaption to application-specific needs with respect to their special terminology and transfer requirements. OpenLogos offers users a set of comfortable customization tools that do not require special knowledge of the system internals. An overview of the possibilities that these tools provide will be presented.     

利用上下文信息的统计机器翻译领域自适应

统计机器翻译系统用于翻译领域文本时,常常会遇到跨领域的问题 当待翻译文本与训练语料来自同一领域时,通常会得到较好的翻译效果;当领域差别较大时,翻译质量会明显下降。某个特定领域的双语平行语料是有限的,相对来说,领域混杂的平行语料和特定领域的单语文本更容易获得。该文充分利用这一特点,提出了一种包含领域信息的翻译概率计算模型,该模型联合使用混合领域双语和特定领域源语言单语进行机器翻译领域自适应。实验显示,自适应模型在IWSLT机器翻译评测3个测试集上均比Baseline有提高,证明了该文方法的有效性。     

Orthographic and morphological processing for English–Arabic statistical machine translation

Much of the work on statistical machine translation (SMT) from morphologically rich languages has shown that morphological tokenization and orthographic normalization help improve SMT quality because of the sparsity reduction they contribute. In this article, we study the effect of these processes on SMT when translating into a morphologically rich language, namely Arabic. We explore a space of tokenization schemes and normalization options. We also examine a set of six detokenization techniques and evaluate on detokenized and orthographically correct (enriched) output. Our results show that the best performing tokenization scheme is that of the Penn Arabic Treebank. Additionally, training on orthographically normalized (reduced) text then jointly enriching and detokenizing the output outperforms training on enriched text.     

Hybrid data-driven models of machine translation

This paper presents an extended, harmonised account of our previous work on combining subsentential alignments from phrase-based statistical machine translation (SMT) and example-based MT (EBMT) systems to create novel hybrid data-driven systems capable of outperforming the baseline SMT and EBMT systems from which they were derived. In previous work, we demonstrated that while an EBMT system is capable of outperforming a phrase-based SMT (PBSMT) system constructed from freely available resources, a hybrid ‘example-based’ SMT system incorporating marker chunks and SMT subsentential alignments is capable of outperforming both baseline translation models for French–English translation. In this paper, we show that similar gains are to be had from constructing a hybrid ‘statistical’ EBMT system. Unlike the previous research, here we use the Europarl training and test sets, which are fast becoming the standard data in the field. On these data sets, while all hybrid ‘statistical’ EBMT variants still fall short of the quality achieved by the baseline PBSMT system, we show that adding the marker chunks to create a hybrid ‘example-based’ SMT system outperforms the two baseline systems from which it is derived. Furthermore, we provide further evidence in favour of hybrid systems by adding an SMT target-language model to the EBMT system, and demonstrate that this too has a positive effect on translation quality. We also show that many of the subsentential alignments derived from the Europarl corpus are created by either the PBSMT or the EBMT system, but not by both. In sum, therefore, despite the obvious convergence of the two paradigms, the crucial differences between SMT and EBMT contribute positively to the overall translation quality. The central thesis of this paper is that any researcher who continues to develop an MT system using either of these approaches will benefit further from integrating the advantages of the other model; dogged adherence to one approach will lead to inferior systems being developed.