Weighted Error Rate for Speech-to-Text Quality Assessment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional word error rate metrics fail to accurately reflect the quality of speech-to-text translation due to equal weighting of all errors, regardless of severity, which is inadequate for assessing the intelligibility of captioned text for deaf or hard of hearing individuals.
Innovation Solution
A translation quality assessment application that assigns weighted coefficients to different types of speech-to-text translation errors based on their impact on comprehension, generating a weighted word error rate that provides a more accurate measurement of translation quality by varying the magnitude of each error's contribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional word error rate metric is used to measure translation accuracy, then the measurement process is simple and fast, but the measurement precision is insufficient because all errors are weighted equally regardless of severity
Solution Approach 1:
The patent applies local quality by assigning different weight coefficients to different types of translation errors based on their severity and impact on comprehension. Critical errors such as wrong entity names, verb tenses, and key information receive higher weights (e.g., 3.0), while minor errors like article usage or preposition choices receive lower weights (e.g., 0.5). This differentiated weighting approach improves measurement precision by reflecting the actual impact of each error type on translation quality, rather than treating all errors equally as in conventional word error rate metrics.
2Reliability
If equal weighting is applied to all translation errors, then the assessment process is straightforward and quick, but the reliability of quality assessment deteriorates because severe errors are not distinguished from minor errors
Solution Approach 1:
The patent implements parameter changes by introducing a weighted coefficient parameter that varies according to error severity. The system defines multiple error categories (critical, major, minor) with corresponding weight coefficients (e.g., 3.0, 1.5, 0.5). This parameter transformation converts the conventional unweighted error count into a reliability-adjusted quality metric, enabling the system to distinguish between transient translation issues and systematic quality problems, thereby improving assessment reliability.
3Measurement precision
If conventional word error rate is calculated, then the computation is simple and fast, but the measurement precision deteriorates because it does not account for the impact of different error types on text intelligibility
Solution Approach 1:
The patent applies preliminary action by pre-defining error categories and their corresponding weight coefficients before the actual translation quality assessment. The system establishes a comprehensive error taxonomy (including critical errors like wrong entity names, major errors like incorrect verb tenses, and minor errors like article usage) with predetermined weights. During assessment, the system only needs to classify errors into these pre-defined categories and apply the corresponding weights, significantly reducing processing time while maintaining high measurement precision for intelligibility assessment.
Data Source
AI summary
According to example configurations, a translation quality assessment application (e.g., system) receives a set of text derived from a translation of an audio signal. The translation quality assessment application aligns and compares the received set of text to reference text to identify occurrence of speech-to-text translation errors in the set of text. For each of the errors, the translation quality assessment application identifies a type associated with the error and retrieves an appropriate weighted coefficient. For example, the translation quality assessment application produces a respective weighted value for each error based on the weighted coefficients. A magnitude of each of the weighted values varies depending on a type of the error that occurred during the translation. The translation quality assessment application utilizes the weighted values to generate a metric indicating a level of quality of the set of captioned text with respect to the set of reference text.