Compressed RNN Weight Matrices for Mobile Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks, particularly recurrent neural networks, face challenges in efficient data processing and storage due to their large computational footprint, making them unsuitable for devices with limited resources like mobile devices.
Innovation Solution
The compression of recurrent neural network weight matrices using a shared projection matrix, reducing the number of parameters and enabling efficient processing and storage, while maintaining performance comparable to uncompressed networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If recurrent neural networks use full-size uncompressed weight matrices, then they achieve high prediction accuracy, but they require large data storage and processing power that mobile devices cannot provide
Solution Approach 1:
The weight matrices are segmented into recurrent weight matrices and inter-layer weight matrices, each handled separately with appropriate compression techniques. This segmentation allows targeted optimization of each matrix type while maintaining overall network performance.
Solution Approach 2:
The patent applies parameter changes by transforming full-rank weight matrices into low-rank representations through decomposition techniques. This changes the parameter dimensionality from O(d²) to O(d×k) where k<<d, reducing storage and computational requirements while preserving essential network functionality.
2Productivity
If recurrent neural networks are compressed to reduce storage and processing requirements, then they can run on mobile devices, but their performance may degrade compared to full-size networks
Solution Approach 1:
The patent merges the compression of recurrent and inter-layer weight matrices through shared low-rank factorization. By combining these compression operations, the network achieves greater overall compression while maintaining performance through coordinated optimization of both matrix types.
Solution Approach 2:
The patent incorporates feedback mechanisms during training to optimize the low-rank decomposition parameters. The training process adjusts the decomposition to minimize performance loss while maximizing compression, ensuring the compressed network maintains prediction accuracy comparable to full-size networks.
3Reliability
If recurrent neural networks use large weight matrices, then they capture complex temporal patterns, but they consume excessive memory and processing resources
Solution Approach 1:
The patent applies dimensionality change by transforming the weight matrices from full-rank d×d representations to low-rank factorizations involving intermediate dimension k. This dimensional transformation reduces storage requirements from O(d²) to O(d×k) while preserving the essential temporal pattern recognition capabilities through the low-rank structure.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a compressed recurrent neural network (RNN). One of the systems includes a compressed RNN, the compressed RNN comprising a plurality of recurrent layers, wherein each of the recurrent layers has a respective recurrent weight matrix and a respective inter-layer weight matrix, and wherein at least one of recurrent layers is compressed such that a respective recurrent weight matrix of the compressed layer is defined by a first compressed weight matrix and a projection matrix and a respective inter-layer weight matrix of the compressed layer is defined by a second compressed weight matrix and the projection matrix. Compression can provide a number of technical advantages, including the ability to process data more efficiently and use less data storage.