Proposition Matching for Machine Translation Quality Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating machine translation quality, whether automated or human-based, often result in inaccurate measurements, leading to unnecessary retraining of machine translation models and wastage of computing resources.
Innovation Solution
The evaluation of machine translation quality is improved by identifying and comparing propositions in source and translated language strings using automated mechanisms or human intelligence, generating a quality score that accurately reflects translation quality and guides model improvements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated metrics (BLEU, METEOR, HTER) are used to evaluate translation quality, then evaluation speed is improved, but measurement accuracy deteriorates
Solution Approach 1:
The patent introduces an intermediary evaluation mechanism that uses proposition extraction and matching as a mediator between source and target texts. This intermediary process transforms the evaluation task into identifying and comparing semantic propositions, which serves as a bridge between automated speed and human-like accuracy in assessing translation quality.
Solution Approach 2:
The patent replaces traditional mechanical string-matching metrics (BLEU, METEOR) with a semantic proposition-based evaluation system. This substitution moves from surface-level lexical comparison to deeper semantic understanding, achieving more accurate translation quality measurement while maintaining automated evaluation capabilities.
2Measurement precision
If human annotators are used to judge translation quality, then measurement accuracy is improved, but subjectivity and disagreement between annotators worsen
Solution Approach 1:
The patent enables the translation evaluation system to assess its own output quality through automated proposition extraction and matching. This self-service mechanism eliminates the need for external human annotators, thereby removing subjectivity and inter-annotator disagreement while maintaining consistent, repeatable evaluation standards.
Solution Approach 2:
The patent implements a feedback loop where the proposition-based evaluation system continuously assesses translation quality and provides measurable feedback. This automated feedback mechanism ensures consistent application of evaluation criteria across all translations, eliminating the variability inherent in human judgment while maintaining high measurement accuracy.
3Reliability
If inaccurate quality measurements are obtained, then unnecessary retraining of models occurs, but computing resources (CPU cycles, memory, network bandwidth, power) are wasted
Solution Approach 1:
The patent performs preliminary proposition extraction and matching to accurately assess translation quality before triggering model retraining. This preliminary evaluation action prevents unnecessary retraining by identifying when translations are actually of sufficient quality, thereby conserving computing resources that would otherwise be wasted on redundant model updates.
Data Source
AI summary
A mechanism is disclosed for evaluating the quality of a machine translation system. Propositions are identified within a source language string and within a translation of the source language string generated by a machine translation system. The propositions can be identified using an automated mechanism or using human translators through the use of a human intelligence task site. Once the propositions have been identified, the propositions identified in the source language string can be compared to the propositions identified in the translated target language string. The results of the comparison can be utilized to compute a quality score for the translation. A final quality score can be generated for the machine translation system by repeating this process for multiple source language strings. The final quality score can then be utilized to improve the quality of models utilized by the machine translation system.