We checked every machine translation. The checks were the part that was wrong.
Languages
Before we released our translation kit, we ran it on a small example app: 70 strings in six formats (JSON, Flutter ARB, Android, Apple .strings, gettext and Markdown) into German, Russian, Japanese, Arabic and Spanish, with a 27-billion-parameter open model on our own machine. Every translated string goes through checks before it is written. Four of the 350 were rejected. All four rejections were wrong.
What the checks look for
A translation that breaks an app is worse than none, so every string must keep its placeholders (%1$s, {name}, {{count}}), its tags, line breaks and URLs; it must be in the target language's script; it must not be empty, runaway or left in English; and plural forms must cover exactly the categories the language has. A string that fails is asked for again with the problem named, and one that fails again is left out, so the app shows English for that one string.
The four rejections
- "Backups" in German was "Backups". The check called it untranslated. German uses the English word. One identical word is often right; two or more identical words are not, so that is the rule now.
- A Russian and a Japanese line with an e-mail address failed the script check: fewer than 60% of the letters were Cyrillic or Japanese, because
[email protected]is written in Latin letters. Addresses are now treated like placeholders: they must come back unchanged, and they do not count as letters. - A Japanese plural message came back as
{count, plural, =0{連続記録なし} other{{count}日連続}}. The branch{連続記録なし}has no spaces, so the placeholder pattern took it for a variable called 連続記録なし. Branch braces in plural messages are structure now, and placeholder names must be ASCII.
Once the checks were fixed, a second run asked the model only for those four strings, since everything else was in the translation memory, and wrote all 350 in 45 seconds.
The part we expected to fail and did not
Plural forms are where naive translation breaks: English has one and other, Russian has one, few, many and other, Arabic six categories, Japanese a single form. We asked for exactly the target's categories and checked each form's placeholders. The model wrote "1 день, 2 дня, 5 дней" for Russian and all six Arabic forms. Every Android plural passed on the first try in all five languages, and so did every plural message except the Japanese one our check misread.
What no check can catch
We then translated the example's 22 web strings into all 42 languages Polyglot knows: 924 of 924 strings passed. Reading through them, one stood out. In Khmer, "You are offline" had come back as a sentence that says "you are online". It is well-formed Khmer in the right script with nothing missing, so every check passed it. Checks catch what breaks an app; a wrong meaning needs a person who reads the language.
The takeaway
Automatic checks on machine output need real output to be tested against. Unit tests with made-up translations passed; a real model writing real Japanese found the gaps in minutes. Polyglot ships with these checks, fixed, and with the tests for each of the four cases.