NLU Model Compression Using Quantization and Perfect Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language understanding (NLU) models in speech processing systems have large memory footprints, leading to high latency and inefficiencies, especially in devices with limited internet access or offline capabilities, where compression techniques are needed to enable local processing of user requests.
Innovation Solution
The implementation of compression techniques for NLU models, including quantization of weights, perfect hashing, and variable-length fingerprint coding, reduces the memory footprint and latency by storing and processing NLU models locally, allowing for efficient execution of user requests without relying on cloud-based systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If NLU models are stored with full precision weights, then predictive performance is maintained, but memory footprint and processing latency increase
Solution Approach 1:
The patent applies quantization to change the precision parameter of model weights from full precision (e.g., 32-bit floating point) to lower precision representations (e.g., 8-bit integers or symbolic representations). This parameter change reduces memory footprint and accelerates processing while the patent maintains predictive performance through techniques like symbolic differentiation and selective precision preservation for critical weights.
2Productivity
If NLU models are compressed for local processing, then latency is reduced and offline capability is enabled, but model accuracy may deteriorate
Solution Approach 1:
The patent applies local quality by differentiating between critical and non-critical weights in the NLU model. Critical weights that significantly impact accuracy are preserved with higher precision or alternative representations, while non-critical weights are aggressively compressed. This selective approach maintains model accuracy for important features while achieving compression for local processing.
Solution Approach 2:
The patent creates compressed copies of the NLU model for local processing. Instead of modifying the original full-precision model, it generates simplified versions using quantization and symbolic representations that can be executed locally with reduced latency while approximating the accuracy of the full model.
3Quantity of substance
If quantization and compression techniques are applied to NLU models, then memory footprint is reduced, but computational complexity of the compression process increases
Solution Approach 1:
The patent performs compression techniques such as quantization and symbolic representation in advance during model training or deployment preparation. The computationally intensive processes of weight analysis, clustering, and symbolic conversion are executed beforehand, producing pre-compressed model artifacts that can then be efficiently loaded and executed without requiring complex real-time compression during inference.
Data Source
AI summary
Devices and techniques are generally described for compression of natural language processing models. A first index value to a first address of a weight table may be stored in a hash table. The first address may store a first weight associated with a first feature of a natural language processing model. A second index value to a second address of the weight table may be stored in the hash table. The second address may store a second weight associated with a second feature of the natural language processing model. A first code associated with the first feature and comprising a first number of bits may be generated. A second code may be generated associated with the second feature and comprising a second number of bits greater than the first number of bits based on a magnitude of the second weight being greater than a magnitude of the first weight.