Speech Recognition Model Training via Multi-Codebook Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for knowledge distillation in speech recognition models require large storage space for embeddings or high computing power for dynamic generation, making them inefficient for devices with limited resources.
Innovation Solution
A method involving multi-codebook quantization to convert floating-point embeddings into quantized codebook data, reducing storage requirements and computational demands by representing embeddings as integers, which are used to train a speech recognition model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If embeddings are stored directly for knowledge distillation, then the representation capability of the student model can be improved, but the storage space occupied increases significantly
Solution Approach 1:
The patent changes the parameter representation of embeddings from high-precision floating-point numbers to low-precision quantized codes. By transforming the data format and precision level, the system maintains the essential representation capability while dramatically reducing storage requirements from gigabytes to kilobytes per audio clip.
Solution Approach 2:
The patent creates a compressed copy of the embedding information through quantized codebooks. Instead of storing the full embedding vectors, the system stores compact codebook indices that can be used to reconstruct approximate embeddings, achieving space-efficient knowledge distillation.
2Volume of stationary object
If embeddings are dynamically generated for knowledge distillation, then the storage space can be reduced, but the computing power required increases significantly
Solution Approach 1:
The patent performs preliminary action by pre-training codebook models offline to create quantized codebooks. This pre-processing step transforms the embedding space into a compressed codebook representation that can be efficiently stored and quickly applied during student model training, avoiding the need for expensive real-time embedding generation.
Solution Approach 2:
The patent introduces quantized codebooks as an intermediary between the teacher model embeddings and the student model. The codebooks serve as a compressed bridge that transfers knowledge from the large teacher model to the small student model without requiring direct access to the full embeddings or expensive dynamic generation.
3Measurement precision
If high-precision floating-point embeddings are used, then the training accuracy can be maintained, but the computational complexity and resource requirements increase
Solution Approach 1:
The patent changes the precision parameter of embedding representation from 32-bit or 64-bit floating-point to 8-bit or lower integer codes. This parameter transformation reduces computational complexity for storage, transmission, and processing while maintaining sufficient accuracy for effective knowledge distillation and student model training.
Data Source
AI summary
A method of training a speech recognition model is provided. The method includes that: speech data of each of a plurality of training samples is inputted into a teacher model and a to-be-trained speech recognition model separately. Additionally, an embedding outputted by the teacher model and encoded data outputted by the to-be-trained speech recognition model are obtained. Furthermore, quantized codebook data is obtained by performing a multi-codebook quantization on the embedding. A loss is calculated based on the encoded data, the quantized codebook data, and text data in the training sample. Moreover, a trained speech recognition model is obtained by stopping training the to-be-trained speech recognition model when the loss is less than or equal to a preset loss threshold and/or trained times is greater than preset trained times.