Joint Decoding Apparatus for Ambiguous Tokenization in Statistical Machine Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional statistical machine translation systems separate tokenization and translation processes, leading to translation errors and suboptimal results due to ambiguous segmentation and limited search spaces, particularly in languages like Chinese and Korean where word boundaries are unclear.
Innovation Solution
A joint decoding apparatus that simultaneously conducts tokenization and translation using a maximum entropy model to generate candidate tokens, calculates tokenization probabilities, processes out-of-vocabulary words, and applies these probabilities to a log-linear model for optimal translation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If tokenization and translation processes are separated, then the system structure is simple and easy to implement, but translation accuracy deteriorates due to segmentation errors and limited search space
Solution Approach 1:
The patent merges the tokenization process and translation decoding process into a unified joint decoding framework. The tokenization decoder and translation decoder share the same search space and probabilistic model, allowing simultaneous optimization of both tokenization and translation. This integration eliminates the interface between separate processes, enabling error reduction through joint probability maximization while maintaining manageable system complexity through shared computational infrastructure.
2Manufacturing precision
If tokenization is performed in pre-processing, then the translation process is straightforward, but translation quality deteriorates due to ambiguous segmentation in languages like Chinese and Korean
Solution Approach 1:
The patent implements dynamic tokenization within the joint decoding framework, where tokenization decisions are not fixed in advance but are dynamically determined during the decoding process based on probabilistic evidence from both the tokenization model and translation model. This allows the system to adapt tokenization boundaries according to contextual information, improving translation quality for languages with ambiguous word boundaries while maintaining reasonable process complexity through efficient search algorithms.
3Manufacturing precision
If a hierarchical phrase structure is used for translation, then the translation process is systematic, but translation accuracy deteriorates when accurate translation rules cannot be applied due to segmentation errors
Solution Approach 1:
The patent applies segmentation at multiple levels within the joint decoding framework, including character-level segmentation for tokenization and phrase-level segmentation for translation. The system segments the source sentence into candidate tokens, then further segments these into translation phrases using hierarchical phrase structure. This multi-level segmentation approach allows the system to handle both fine-grained tokenization issues and coarse-grained translation issues, improving accuracy while managing complexity through hierarchical organization.
Data Source
AI summary
Disclosed are a joint decoding apparatus and a joint decoding method that joins a tokenization process and a translation process. More particularly, the present disclosure can generate all available candidate tokens, reduce translation errors, and obtain an optimal translation result by jointly conducting a decoding by simultaneously conducting the tokenization process and the translation process for an input character sequence while decoding.