Automated Voice Query Model Training via User Behavior Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual supervision is inefficient for continuously updating and training Automatic Speech Recognition (ASR) systems due to the rapid addition and removal of content in content distribution networks, making it difficult to maintain accurate voice query recognition.
Innovation Solution
An automated method for evaluating ASR transcriptions and providing annotations based on user behavior and session analysis, identifying erroneous transcriptions and patterns of query reformulation to automatically determine corrections, which are then used to improve the machine learning model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual training is used for ASR systems, then training quality can be maintained through human supervision, but the system cannot keep up with the rapid rate of content addition and removal in content distribution networks
Solution Approach 1:
The system enables ASR models to self-train by automatically generating training data from user interactions. The system collects voice queries, generates transcriptions, determines outcomes based on user behavior, and uses this data to retrain models without human intervention, allowing continuous adaptation to new content at network scale
Solution Approach 2:
The system implements feedback loops where ASR transcriptions are evaluated against actual user behavior outcomes. Correct transcriptions are identified through positive outcomes (successful content playback) and incorrect ones through negative outcomes (user corrections or lack of playback), creating labeled training data that feeds back into model improvement cycles
2Productivity
If automated methods are used to continuously update ASR systems, then training data can be generated at scale without manual intervention, but the quality and accuracy of transcriptions may deteriorate
Solution Approach 1:
The system uses user behavior as feedback to validate transcription accuracy. Positive outcomes (content plays successfully) indicate correct transcriptions, while negative outcomes (user repeats query or no playback) indicate errors. This automated feedback mechanism maintains quality standards at scale
Solution Approach 2:
The system self-corrects by identifying erroneous transcriptions through outcome analysis and automatically generating corrected training examples. The model retrains on these self-generated corrections, continuously improving accuracy without human supervision
3Reliability
If traditional supervised training is used, then model accuracy can be maintained, but the complexity and time required for retraining increases with each content update
Solution Approach 1:
The system implements continuous training where models are constantly retrained on new data from user interactions rather than periodic batch updates. This continuous learning process adapts to content changes in real-time, maintaining accuracy without lengthy retraining cycles
Solution Approach 2:
The system automatically collects, labels, and processes training data from production traffic, eliminating the need for manual data preparation and model retraining. The entire training pipeline operates autonomously, dramatically reducing the time investment required for continuous model improvement
Data Source
AI summary
Methods for automatically evaluating ASR outputs and providing annotations, including corrections, on the transcriptions—in order to improve recognition—may be based on an analysis of sessions of user voice queries, utilizing time-ordered ASR transcriptions of user voice queries (i.e., user utterances). This utterance-based approach may involve extracting both session-level and query-level characteristics from a voice query sessions and identifying patterns of query reformulation in order to detect erroneous transcriptions and automatically determine an appropriate correction. Alternative, or in addition, ASR outputs may be evaluated based on user behavior. The outcomes may be classified as positive or negative. An ASR transcription may be labeled using the description of the outcome. The labeled transcription may be used as training data to train a model to output improved transcriptions of voice queries.