09 · Human Parity

In March 2018 Microsoft Research announced that its machine had reached human parity translating Chinese news into English, and within the same year two teams working independently showed that the word parity was a product of the evaluation method, not a property of the translation.

The claim was not weak, and it deserves its full force before it is broken. Hany Hassan and colleagues published arXiv:1803.05567, "Achieving Human Parity on Automatic Chinese to English News Translation," reporting that their neural system's output on the WMT 2017 Chinese-to-English news test set was statistically indistinguishable from the work of professional human translators. The evidence came in two layers. The first was automatic metrics. The second was a formal protocol in which bilingual judges scored the output. Nothing offhand about it, and both layers said the same thing.

Samuel Läubli, Rico Sennrich and Martin Volk took the same translations back to human judges, changing one variable — the raters read whole documents instead of isolated sentences served in shuffled order. The paper, "Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation," appeared at EMNLP 2018, and its result reads: "human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences." Same system, same translations, human judges in both conditions. The only thing that changed was the length of what was put in front of them.

The errors that surface over a longer read have names of their own. Dropped pronouns. Coherence that breaks between sentences. One term called two different things in the same story. None of the three is visible in an isolated sentence, because an isolated sentence gives them nowhere to appear. A sentence cut out of a paragraph has nothing for a pronoun to point back to, and no second occurrence for a term to contradict its first.

Antonio Toral, Sheila Castilho, Ke Hu and Andy Way checked the same claim by another route. Their paper, "Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation," at WMT 2018, found that the parity result depended on how the evaluation had been designed, and named two choices that flattered the machine directly — the source text used for testing was itself translated material rather than text originally authored in that language, and the raters were shown no inter-sentential context.

The sentence-sized window is sixteen years older than this episode. Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu proposed BLEU at ACL 2002 as a modified n-gram precision score against reference translations, with a brevity penalty to stop a system scoring by writing short, and its own authors called it an "understudy to skilled human judges." An understudy, not the lead. The word was in the founding paper from 2002, and the understudy then held the stage for roughly two decades.

BLEU cannot report what breaks across sentences, because it never looks at anything longer than a sentence. This is not a flaw in the formula. It is the size of the window. Chris Callison-Burch, Miles Osborne and Philipp Koehn wrote at EACL 2006 that a rise in BLEU is "neither necessary nor sufficient" for a real improvement in quality, with counterexamples in both directions: systems BLEU ranked higher and humans ranked lower, and systems where it ran the other way. Twelve years before the 2018 announcement, the warning was already in the field's own proceedings.

Human judges fail the same way if given the same window. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan and Wolfgang Macherey published in TACL 9 in 2021 the largest MQM study yet conducted, having professional translators read with full document context and re-score the WMT2020 systems in two language pairs. The outcome was "a substantially different ranking of evaluated systems from the one established by the WMT crowd workers." The fault, then, is not whether the judge is a machine or a person. It is what the judge was shown.

The line runs back to 1966. ALPAC was established in April 1964, chaired by John R. Pierce of Bell Telephone Laboratories with six other members. Its report, Language and Machines — Computers in Translation and Linguistics, was issued in November 1966. It defined machine translation very narrowly, as translation "by algorithm from machine-readable source text to useful target text, without recourse to human translation or editing," and concluded under that definition that "there has been no machine translation of general scientific text, and none is in immediate prospect." Funding was cut, and MT research in America stopped for roughly twenty years.

The two cases run the same mechanism with the sign reversed. In 1966 a narrow definition reduced an entire field's output to zero. In 2018 a narrow protocol raised one set of translations to human parity. Every generation's evaluation regime measures what it happens to be able to measure, and what it can measure then becomes the meaning of the word good. One detail from the ALPAC report itself is worth noting: one of the three things it recommended funding was translation evaluation methods.

The field has fixed this, and it fixed it by widening the window rather than by waiting for the models to improve. Maja Popović proposed chrF at WMT 2015, scoring on character n-grams instead. Ricardo Rei and colleagues proposed COMET at EMNLP 2020, learning directly from human judgments rather than counting matching n-grams. The WMT24 organizers themselves name the two top performing automatic metrics as MetricX-23-XL and CometKiwi-DA-XL — BLEU is no longer the yardstick by which the field's best systems are ranked. WMT25 went further, to 32 language pairs and 60 systems, adding a new Literary domain built from documents of roughly 5,000 words, for the reason printed in the report's own title — "Time to Stop Evaluating on Easy Test Sets."

The Thai-side number cannot be confirmed, and this book will not guess it. NLLB-200 reports a 44% average BLEU improvement over the prior state of the art across more than 40,000 translation directions on FLORES-200, which spans 204 language and script pairs, and Thai (tha_Thai) is among them. But a BLEU or chrF score specific to Thai could not be traced to a primary source in this research pass. As for the WMT25 system tables, all 32 language pairs of them, inspection turned up no English-Thai pair among them. The wider window has not yet reached the language this book is written in.

What the sentence-sized window cannot see is the set of obligations that arise only once a text runs longer than one sentence: the pronoun that has to point back to a person named three sentences ago, the term that has to be called the same thing every time until the end, and the register that says who is speaking to whom. The last of those has evidence of its own. Luis Cihlar and colleagues examined the automatic translation of Korean honorifics and reported at the IAAI workshop at LREC 2026 that models shift politeness level according to whether the addressee is stated explicitly or has to be inferred. The systems match a cue at the surface of the text; they do not model the relationship. A single sentence has no addressee. A whole text does.

What could not cross: The pronoun that has to point across a sentence boundary, the term that has to stay fixed for the length of a story, and the register that specifies who is speaking to whom — all three are obligations of a whole text, not of a single sentence. An instrument with a one-sentence window therefore reports that nothing is missing, because it never looks at the place the missing thing was.

Overturned if: Take Hassan et al. 2018's translations back to human judges under both conditions, shuffled isolated sentences and full documents. If the raters do not prefer the human translation more strongly when reading full documents, or if Läubli, Sennrich and Volk 2018's result fails to replicate in other language pairs, this verdict fails.

Sources: Hassan et al., "Achieving Human Parity on Automatic Chinese to English News Translation," arXiv:1803.05567 (2018) — https://arxiv.org/abs/1803.05567 · Läubli, Sennrich, Volk, "Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation," EMNLP 2018 — https://aclanthology.org/D18-1512/ · Toral, Castilho, Hu, Way, "Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation," WMT 2018 — https://aclanthology.org/W18-6312/ · Papineni, Roukos, Ward, Zhu, "BLEU: a Method for Automatic Evaluation of Machine Translation," ACL 2002 — https://aclanthology.org/P02-1040/ · Callison-Burch, Osborne, Koehn, "Re-evaluating the Role of Bleu in Machine Translation Research," EACL 2006 — https://aclanthology.org/E06-1032/ · Popović, "chrF: character n-gram F-score for automatic MT evaluation," WMT 2015 — https://aclanthology.org/W15-3049/ · Rei et al., "COMET: A Neural Framework for MT Evaluation," EMNLP 2020 — https://aclanthology.org/2020.emnlp-main.213/ · Freitag, Foster, Grangier, Ratnakar, Tan, Macherey, "Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation," TACL 9, 2021 — https://aclanthology.org/2021.tacl-1.87/ · Pierce, Carroll, et al., Language and Machines — Computers in Translation and Linguistics (ALPAC Report), National Academy of Sciences, 1966 — http://www.mt-archive.info/ALPAC-1966.pdf · Kocmi et al., "Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet," WMT 2024 — https://aclanthology.org/2024.wmt-1.1/ · Kocmi et al., "Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets," WMT 2025 — https://aclanthology.org/2025.wmt-1.22/ · Kocmi et al., "Preliminary Ranking of WMT25 General Machine Translation Systems," arXiv:2508.14909 (2025) — https://arxiv.org/abs/2508.14909 · NLLB Team et al., "No Language Left Behind: Scaling Human-Centered Machine Translation," arXiv:2207.04672 (2022) — https://arxiv.org/abs/2207.04672 · Cihlar, Bui, Park, Mager, Bisang, von der Wense, "Investigating the Automatic Translation of Korean Honorifics," LREC 2026 IAAI Workshop — https://aclanthology.org/2026.iaai-1.3/