Automatic Translation Evaluation

Research into automatic correction of human translation wins prize

Natural language processing (NLP) scientists from Lilt and the University of California at Berkeley published a NAACL award winning research paper on June 17, 2022. The paper introduced a Translation Error Correction (TEC) model that automates the correction of human translations.

The scientists told Slator, “The aim of this work is to replicate the sentence-level corrections made by expert linguists when reviewing the translations written by other linguists. While each language pair has its own nuance, the approach we take is handling is quite common.”

They added: “Just as neural machine translation applies effectively to many language pairs, we expect this approach to translation error correction to be widely applicable as well, and we look forward to reporting our future progress as we extend this work to other areas.” language pairs.”

The TEC model uses the same structural foundation as: Automatic Post Processing (APE), which has been widely studied but differs from TEC in many ways. For example, TEC uses human-made errors as data and focuses on error correction rather than detection. TEC can also distinguish content that does not require editing.

Unlike TEC, APE is “dominated by the fluency errors typical of MT systems (74% of sentences),” the article said, adding that the “TEC corpus shows a wider distribution of errors that human translators tend to to make .”

Asked to define the translation fluency, the scientists replied, “Fluid fluency of a translation describes whether a native speaker of the language would use the formulation, structure, and choice of words that appear in the translation.”

Input from 10 human translators

Scientists used a bilingual corpus called ACED, which contains three datasets from different domains. The data consists of 35,261 English-German translations performed and edited by professional translators (not post-edited).

To prepare the data, the scientists removed duplicate source sentences, deleted translations rewritten by reviewers, and classified errors into three main categories: monolingual edits found in the target text, bilingual edits that correct translation errors, and preferred edits.

The ACED data has been pre-processed, trained and refined using real human corrections. Tests and comparisons were made between TEC and other models, including MT, GEC (grammatical error correction), and BERT-APE

Nine professional translators participated in the study as reviewers to determine the real applicability of the model. The nine were asked to revise sentences (of which 255 had suggestions for corrections) and provide qualitative observations.

A tenth professional translator was tasked with reviewing the reference translations in the data set and the quality of the sentences assessed by the other nine.

Slator Software-as-a-Service (SaaS) Localization Report - Product Image

Slator Software-as-a-Service (SaaS) Localization Report

90-page report on how SaaS companies are partnering with LSPs to localize products and drive market/user growth. incl. market size, technical assessment.

Next step in translation workflow automation?

Comparisons with other models revealed significant differences in how the TEC model ultimately performed. For example,

  • the professional reviewers accepted 79% percent of the TEC suggestions for correction;
  • reviewers spent less time grading when suggestions were accepted; and
  • domain customization proved to be critical to performance – and customization, essential for correcting translation errors.

Five of the nine reviewers emphasized the need for reliability. In the test, some suggestions were incorrect or the system failed to perform a reliable operation.

Three reviewers felt that the TEC system “could act as a mnemonic or substitute for researching customer-specific requirements.”

Three reviewers noted that TEC could help “by making them aware of the mistakes to watch out for, especially in repetitive content where it can be easy to overlook details.”

Given the findings, TEC could be the next step in automating the translation workflow. The greater the precision of the model, the greater its potential to make a practical difference during the translation production review stages.

Leave a Comment

Your email address will not be published.