Federated Learning for Spoken Language Understanding Models
Solution Overview
Problem
In natural language processing systems, obtaining annotated data for machine learning models without centralized annotation workflows is challenging, especially for complex output spaces, which impedes model improvements and user experience due to unsatisfactory responses.
Innovation Solution
A system that uses a federated learning framework to solicit inputs from users, identifies needed output labels, and updates on-device machine learning models, minimizing the need to share user data by generating labeled input data locally and sending updates to a remote system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized annotation workflows are used to obtain annotated data for machine learning models, then the quality and quantity of training data can be improved, but the complexity of the system increases and user data privacy is compromised
Solution Approach 1:
The system enables users to automatically provide annotated training data through their natural interactions with the device. Users inadvertently annotate data by their spoken corrections and follow-up statements, eliminating the need for centralized annotation workflows while maintaining data quality.
Solution Approach 2:
The patent introduces an intermediary mechanism that captures user corrections and transforms them into annotated training data. This intermediary process occurs locally on the device, bridging the gap between user interactions and model training without requiring centralized annotation infrastructure.
2Quantity of substance
If centralized annotation workflows are implemented, then comprehensive training data can be collected, but data privacy concerns increase and user trust decreases
Solution Approach 1:
The system processes and annotates data locally on individual devices rather than centralizing data collection. Each device independently captures and processes user corrections, keeping sensitive information local and minimizing privacy risks while still accumulating training data across the user base.
Solution Approach 2:
Users automatically generate annotated training data through their natural interactions without needing to consciously share personal information. The system self-annotates data from user corrections, building training datasets while maintaining user privacy and trust.
3Reliability
If complex centralized annotation systems are used, then model performance can be improved, but the time and resources required for data collection increase
Solution Approach 1:
The system continuously collects annotated training data through ongoing user interactions rather than requiring separate, time-intensive annotation campaigns. Every user correction and follow-up statement automatically contributes to the training dataset, making data collection an continuous byproduct of normal device usage.
Solution Approach 2:
The system automatically annotates training data from user interactions without requiring manual intervention or dedicated annotation resources. User corrections and follow-up statements are automatically processed into labeled training examples, eliminating the need for separate annotation workflows and reducing data collection time.
Data Source
AI summary
Techniques for using a federated learning framework to update machine learning models for spoken language understanding (SLU) system are described. The system determines which labeled data is needed to update the models based on the models generating an undesired response to an input. The system identifies users to solicit labeled data from, and sends a request to a user device to speak an input. The device generates labeled data using the spoken input, and updates the on-device models using the spoken input and the labeled data. The updated model data is provided to the system to enable the system to update the system-level (global) models.