Semi-supervised Word Alignment via Discriminative Re-ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current statistical machine translation (SMT) systems for word alignment in multiple languages lack accuracy due to the limitations of the Expectation-Maximization algorithm and the need for further improvements in aligning words across parallel documents.
Innovation Solution
A system and method for aligning words in parallel segments by calculating a first probability distribution of word alignments using a model estimate, modifying it based on annotated parallel segments, and discriminatively re-ranking sub-models to generate a second probability distribution, which enhances the accuracy of word alignments through iterative refinement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the Expectation-Maximization algorithm is used for word alignment, then word alignments can be induced in large corpora, but the accuracy of word alignments is insufficient
Solution Approach 1:
The patent segments the word alignment problem into multiple hierarchical levels: sentence-level alignment, word-level alignment, and phrase-level alignment. By dividing the large corpus into manageable segments and processing them through multiple passes with different granularities, the system achieves higher accuracy without overwhelming computational complexity
Solution Approach 2:
The patent performs preliminary sentence alignment and segmentation before conducting detailed word alignment. By pre-identifying parallel sentence structures and potential alignment candidates, the system reduces the search space for the E-M algorithm, improving both accuracy and efficiency
2Measurement precision
If manual annotations are used to correct word alignments, then alignment accuracy improves, but the processing time and computational resources increase
Solution Approach 1:
The patent applies manual annotations selectively to only those alignment cases where automated methods are uncertain or likely to fail. By focusing human expertise on borderline cases rather than processing entire corpora manually, the system achieves accuracy improvements with minimal time investment
Solution Approach 2:
The system implements an iterative feedback loop where manual correction results are fed back into the training data, continuously improving the automated alignment model. This allows the system to progressively reduce its dependence on manual annotations over time while maintaining high accuracy
Data Source
AI summary
A system and method for aligning words in parallel segments is provided. A first probability distribution of word alignments within a first corpus comprising unaligned word-level parallel segments according to a model estimate is calculated. The model estimate is modified according to the first probability distribution. One or more sub-models associated with the modified model estimate are discriminatively re-ranked according to word-level annotated parallel segments. A second probability distribution of the word alignments within the first corpus is calculated according to the re-ranked sub-models associated with the modified model estimate.