Measuring Unintended Memorization in ASR Models via Sped-Up Canary Utterances
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is currently no efficient technique available to measure unintended memorization of utterances used for training automatic speech recognition (ASR) models.
Innovation Solution
A computer-implemented method is introduced that involves obtaining a pre-trained ASR model, creating a set of canary speech utterances, speeding up these utterances, fine-tuning the ASR model on the sped-up canary speech utterances, and measuring unintended memorization based on speech recognition results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing techniques are used to measure unintended memorization in ASR models, then measurement capability is achieved, but computational complexity and resource requirements increase significantly
Solution Approach 1:
The patent extracts the essential measurement function by using only the pre-trained ASR model itself to evaluate memorization, eliminating the need for additional reference models. The canary utterances are designed to be recognizable only if memorized, allowing direct measurement without external references.
Solution Approach 2:
The ASR model serves itself by using its own pre-trained parameters to recognize canary utterances and measure memorization. The model's inherent capabilities are leveraged to detect whether it has memorized training data, eliminating dependency on external evaluation systems.
2Productivity
If canary utterances are created and sped up for measurement, then measurement efficiency is improved, but the complexity of the measurement process increases
Solution Approach 1:
Canary utterances are pre-generated from random text sequences using text-to-speech synthesis before the measurement process begins. These utterances are designed with unique characteristics (random consonants, specific pacing) that make them identifiable only if memorized, preparing the measurement substrate in advance.
Solution Approach 2:
The canary utterances are sped up by a factor of 4x compared to normal speech rate. This parameter change creates a distinctive temporal pattern that the ASR model can recognize if it has memorized the utterances during training, providing a clear measurement signal.
Data Source
AI summary
A method includes obtaining an automatic speech recognition (ASR) model pre-trained on an initial training dataset, creating a set of canary speech utterances, and speeding up each canary speech utterance in the set of canary speech utterances. The operations also include fine-tuning the ASR model on the set of sped-up canary speech utterances and measuring un-intended memorization of the fine-tuned ASR model based on speech recognition results performed by the fine-tuned ASR model on the sped-up canary speech utterances.