Methodologies for Measuring the Correlations between Post-Editing Effort and Machine Translatability

Against the background of a wider research project that aims to investigate the correlation, if any, between post-editing effort and the presence of negative translatability indicators in source texts submitted to Machine Translation (MT), this paper sets out to assess the potential of two methods for measuring the effort involved in post-editing MT output. The first is based on the use of the keyboard-monitoring program Translog; the second on Choice Network Analysis (CNA). The paper reviews relevant research in both machine translatability and MT post-editing, and appraises, in particular, the suitability of think-aloud protocols in assessing post-editing effort. The combined use of Translog and CNA is proposed as a way of overcoming some of the difficulties presented by the use of think-aloud protocols in the current context. Initial results from a study conducted at Dublin City University confirm that triangulating data from Translog and CNA can cast light on the temporal, cognitive and technical aspects of post-editing effort.