1,082 translations had a word in the wrong alphabet. Our checks passed every one.
Languages
On 31 October we found a Hungarian sentence on one of our sites with Chinese words in the middle of it. We searched the stored translations of every site we run for letters that do not belong to their language and found 1,082 translations like it, 581 of them on published pages. Every one had passed our checks.
What we found
All of it was written by the local models that translate our sites, and all of it is the same slip: a word or two in another language. From the stored translations, and from those refused since:
- Chinese inside other languages: "they are" as 它们是 in the middle of a Thai sentence; the Chinese for "track", 跟踪, given a Hungarian ending ("támadók跟踪ására"); similar words in Polish and Greek.
- Russian inside Greek, Armenian, Hebrew and Slovenian: "full house" in Greek came out as "φουλ хаус", the second word in Cyrillic.
- Look-alike letters: a Cyrillic "о" or "а" at the end of a Finnish or Czech word. Nobody reading the page would notice; a search engine or a screen reader would.
- Neighbouring scripts: a Bengali letter inside a Kannada word, Devanagari inside Kannada.
- Whole lines in Thai on Khmer pages: titles, labels and table cells on two sites' Khmer pages were simply written in Thai.
Our store had seven such lines of its own. In one, "planted values" came out in Romanian as "valori植入".
Why the checks passed them
The script check asked whether at least 45% of a translation's letters were in its language's script. That catches English left in place, which is what it was built for. Two Chinese letters in a Hungarian sentence of a hundred letters do not change a share, and a single Cyrillic letter changes it even less.
Short text had a different rule. Under 25 letters of English, the check only made sure the answer had not stayed in English. A Khmer table cell written in Thai contains no English at all, so it passed.
The rule that catches them
A letter that belongs to a script the language does not use, and that the English source does not contain, is a mistake. Three exceptions keep the rule from crying wolf:
- Latin letters are always allowed: brand names, code and units appear in every language.
- Chinese, Japanese and Korean are not checked, because Japanese mixes kana and kanji and Korean can use hanja.
- Serbian and Punjabi, each written in two scripts, are checked only for Chinese, Japanese and Korean letters.
We ran the rule over everything our sites had stored, and every line it flagged that we read was wrong. Since 31 October it has refused 1,652 translations as they were made, 1,549 of them Khmer: the model has not stopped writing Thai for Khmer, but now it is asked again instead of published.
Cleaning up
Stored translations are checked again only when their language is translated again, so every affected language was translated again: 61 of them across eleven sites. Two lines came back with the same mistake a second time: the Romanian "valori植入" and a Greek line with 面向 in it. We wrote those two by hand. On 2 November no published page of ours had a letter of another script.
Where the rule lives now
It is in translation-checks 1.1.0, free on GitHub, next to a new check that every number comes back and knows that "3 million" is 300万 in Japanese. Run against our sites' 568,574 accepted translations, the new checks raised no false alarm. Polyglot 1.1.0 ships them too. The first time our checks were the problem is in an earlier note.