ASR Session Metric for Error Rate Prediction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current automatic speech recognition (ASR) dictation systems lack effective confidence scoring mechanisms to provide timely feedback and corrective actions to users when recognition performance degrades, relying heavily on human intervention and not adequately addressing individual user variations in system updates.

Innovation Solution

A post-processing module is introduced to calculate a session-level metric correlated with verbatim recognition error rates, using a word-level confidence metric based on logistic regression, which compares performance to a baseline to warn users and recommend actions such as verifying audio inputs or text results, allowing for real-time or offline adjustments.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If confidence scoring is used to identify incorrectly recognized words, then recognition accuracy can be improved, but it cannot provide timely feedback and corrective actions when system performance degrades

Engineering Contradiction:

Improverecognition accuracyVSAvoidfeedback timing

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary analysis by computing session-level metrics from confidence scores during the dictation session itself, before degradation becomes apparent to the user. This allows the system to detect performance issues in advance and provide timely warnings and corrective actions rather than waiting for user complaints or manual verification.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If manual verification of recognition results is required, then recognition accuracy can be improved, but user productivity decreases due to additional manual work

Engineering Contradiction:

Improverecognition accuracyVSAvoiduser efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system implements automated feedback by monitoring session-level metrics derived from confidence scores and comparing them against thresholds. When degradation is detected, the system automatically warns users and suggests corrective actions, eliminating the need for continuous manual verification while maintaining high recognition accuracy through proactive quality control.

Inventive Principle:

Principle #23Feedback

3Reliability

If system updates are applied to improve recognition performance, then overall system capability is improved, but individual user performance may degrade due to variations in user speech patterns

Engineering Contradiction:

Improvesystem capabilityVSAvoiduser-specific performance

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system applies local quality control by computing session-level metrics for each individual user and comparing them against user-specific baselines. This allows the system to detect when a general system update causes degradation for a particular user, and to provide targeted warnings and corrective actions tailored to that user's speech patterns and performance characteristics.

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS9269349B2Automatic methods to predict error rates and detect performance degradation

Publication Date: 2016.02.23 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

An automatic speech recognition dictation application is described that includes a dictation module for performing automatic speech recognition in a dictation session with a speaker user to determine representative text corresponding to input speech from the speaker user. A post-processing module develops a session level metric correlated to verbatim recognition error rate of the dictation session, and determines if recognition performance degraded during the dictation session based on a comparison of the session metric to a baseline metric.