Deep Neural Network Parameter Reduction via Low-Rank Matrix Factorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) used in automatic speech recognition (ASR) face challenges with adaptation and personalization due to a large number of parameters, making them costly and inefficient for deployment across multiple individual speakers.
Innovation Solution
Applying low-rank factorization to the original weight matrix of the DNN model, converting it into smaller matrices, and storing speaker-specific parameters in a square matrix initialized as an identity matrix, which is then updated to adapt the model, reducing the overall number of parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used to improve ASR accuracy, then modeling capability is improved, but the number of parameters increases making adaptation and personalization challenging
Solution Approach 1:
The patent segments the large weight matrix into multiple smaller sub-matrices through low-rank factorization. The original weight matrix W is decomposed into W = A × B, where A and B are smaller matrices. This segmentation reduces the number of parameters that need to be stored and adapted while maintaining the modeling capability of the DNN for accurate ASR.
Solution Approach 2:
The patent changes the parameter representation by transforming the full-rank weight matrix into a low-rank factorized form. By representing the weight matrix as a product of two smaller matrices with lower ranks, the number of parameters is reduced from O(n²) to O(nk) where k is the rank parameter, making adaptation and personalization feasible.
2Measurement precision
If DNN models are personalized for multiple individual speakers, then ASR accuracy for each speaker is improved, but storage cost increases due to needing to store large DNN models for each speaker
Solution Approach 1:
The patent segments the speaker-specific adaptation parameters into a small set of low-rank factors. Instead of storing complete personalized weight matrices for each speaker, only the small factor matrices need to be stored. This reduces storage requirements from O(n²) per speaker to O(nk) per speaker, making multi-speaker personalization cost-effective.
Solution Approach 2:
The patent creates a shared base DNN model that can be copied and adapted for multiple speakers. The low-rank factorized representation allows the same base model to be efficiently personalized for different speakers by updating only the small factor matrices, rather than storing complete separate models for each speaker.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The adaptation and personalization of a deep neural network (DNN) model for automatic speech recognition is provided. An utterance which includes speech features for one or more speakers may be received in ASR tasks such as voice search or short message dictation. A decomposition approach may then be applied to an original matrix in the DNN model. In response to applying the decomposition approach, the original matrix may be converted into multiple new matrices which are smaller than the original matrix. A square matrix may then be added to the new matrices. Speaker-specific parameters may then be stored in the square matrix. The DNN model may then be adapted by updating the square matrix. This process may be applied to all of a number of original matrices in the DNN model. The adapted DNN model may include a reduced number of parameters than those received in the original DNN model.