Repository navigation
Keep words and word order when repairing translated input names - #1504
Merged
Merged
Conversation
The mention repair that runs after machine translation read `$name` plus any letters after it as one mangled name, so in Japanese and Chinese it deleted the words that follow an input. It also rewrote every input into English order whenever one name looked wrong, swapping inputs a translation had correctly reordered. Now only the ASCII name is replaced, the text after it is kept, and only names the source lacks are repaired. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
After machine translation,
repairMentionsPositional(protect.ts) puts back any template input the model mangled. It had two defects, and neither one fails any check, because the result still contains every input and parses fine:$nameplus any letters after it as one name. In a language written without spaces,$permissionをブロックしたためwas "repaired" to$permission, deleting everything up to the next punctuation mark. This affected Japanese heavily, and Chinese and Cantonese wherever an input touched the next word.$lowerbefore$highercame out saying the reverse. This affected every language with a different word order.Now only the ASCII name the tokenizer reads is treated as the input, and the text after it is kept. A translation that uses exactly the source's names is left alone, whatever their order. Only a name the source doesn't have is replaced, and only in its own position.
The strings already damaged are repaired separately in the locale-repair PR.
Related issues
Verification
translate.test.ts: words after a name are kept, a misspelled name is fixed without losing the words after it, reordered inputs are kept, and only the mangled name is repaired, wherever it sits.npm run check:now,npm run test:run.Checklist
🤖 Generated with Claude Code