Trace Norm Regularization for Embedded Speech Recognition Model Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current speech recognition systems face challenges in achieving accurate results on devices with limited processing power and memory, particularly in handling continuous speech and large vocabularies, due to compute and memory constraints.
Innovation Solution
The implementation of trace norm regularization and efficient kernel optimization for low-batch-size inference on ARM processors, which allows for the training of low-rank models that reduce parameter size and computation requirements without sacrificing accuracy, using techniques like truncated singular value decomposition and custom assembly kernels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If larger neural network models are used for speech recognition, then accuracy is improved, but device complexity and resource requirements worsen
Solution Approach 1:
The patent applies trace norm regularization to change the parameter distribution in neural network weight matrices, encouraging low-rank structures. This transforms the model to use fewer effective parameters while maintaining accuracy, directly resolving the contradiction between accuracy and model complexity
Solution Approach 2:
The patent extracts and eliminates redundant parameters through low-rank factorization and trace norm regularization. By removing unnecessary parameters while preserving the essential information in weight matrices, the system achieves reduced model size without sacrificing recognition accuracy
2Measurement precision
If more parameters are used in the neural network, then speech recognition performance is improved, but memory requirements worsen
Solution Approach 1:
The patent changes the parameter representation by applying trace norm regularization, which modifies the weight matrix structure to be low-rank. This reduces the number of stored parameters while maintaining the same computational functionality, thereby reducing memory requirements without sacrificing performance
Solution Approach 2:
The patent reduces the dimensionality of weight matrices through low-rank factorization. By representing high-dimensional weight matrices as products of lower-dimensional matrices, the system achieves the same computational effect with fewer parameters, directly reducing memory footprint
3Measurement precision
If higher precision computation is used, then inference accuracy is improved, but processing speed and power efficiency worsen
Solution Approach 1:
The patent changes the computational approach by using trace norm regularized weight matrices that can be efficiently computed with lower precision. The low-rank structure enables faster matrix operations that are less computationally intensive, achieving a balance between inference accuracy and processing speed
Solution Approach 2:
The patent applies partial precision computation by using optimized low-rank matrix operations that require fewer computational resources than full precision would suggest. By leveraging the low-rank structure, the system achieves sufficient accuracy with reduced computational effort, improving speed and power efficiency
Data Source
AI summary
Described herein are systems and methods for compressing and speeding up dense matrix multiplications as found, for examples, in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, trace norm regularization technique embodiments were introduced and studied for training low rank factored versions of matrix multiplications. Compared to standard low rank training, the methods more consistently lead to good accuracy versus number of parameter trade-offs and can be used to speed-up training of large models. Faster inference may be further enabled on ARM processors through kernels optimized for small batch sizes, resulting in speed ups over the currently used library. Beyond LVCSR, the techniques are also generally applicable to embedded neural networks with large fully connected or recurrent layers.