Compressed Word Embeddings via Auto-Encoder for NLP Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language processing systems face challenges in storing and processing large word embeddings due to memory limitations in electronic devices, as they are typically represented as large matrices of real numbers, making it difficult to store and process efficiently.
Innovation Solution
The implementation of a natural language processing system that employs compressed word embeddings using an auto-encoder processing unit with encoder and decoder circuitry, where word embeddings are compressed into binary vectors, allowing for efficient storage and decompression, enabling the system to operate independently with limited memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If word embeddings are stored as large matrices of real numbers, then the system maintains high processing accuracy, but the memory storage requirement increases significantly
Solution Approach 1:
The patent transforms word embeddings from real number format to binary format, fundamentally changing the data representation parameter. This allows the system to store embeddings more efficiently while maintaining functionality through learned binary representations that can be decompressed back to approximate real number values
Solution Approach 2:
The patent introduces an autoencoder architecture as an intermediary mechanism between storage and processing. The encoder compresses real number embeddings to binary form for storage, while the decoder reconstructs them back to real numbers for processing, effectively mediating between the conflicting requirements of storage efficiency and processing accuracy
2Quantity of substance
If compressed binary embeddings are used, then memory storage is reduced, but the system complexity increases due to encoder-decoder architecture
Solution Approach 1:
The autoencoder architecture is trained to be self-sufficient in handling the compression and decompression tasks. Once trained, the encoder and decoder work autonomously to transform between binary and real number representations without requiring external intervention, making the added complexity self-managing rather than externally imposed
3Speed
If uncompressed word embeddings are stored, then processing speed is maintained, but the electronic device requires more memory capacity
Solution Approach 1:
The system performs preliminary compression of word embeddings to binary format before storage. When processing is needed, the decoder quickly reconstructs the embeddings from the compressed binary form, maintaining processing speed while having efficiently used storage space during the interim period
Data Source
AI summary
Described herein are systems and methods that provide a natural language processing system (NLPS) that employs compressed word embeddings. An auto-encoder that includes encoder circuitry and decoder circuitry can be used to produce the compressed word embeddings. The decoder circuitry is trained to decompress the word embeddings with reduced or minimal differences between the original uncompressed word embeddings and the corresponding decompressed word embeddings. One or more parameters of the trained decoder circuitry are transferred to the NLPS, where the NLPS is then trained using the compressed word embeddings to improve the correctness of the responses or actions determined by the NLPS.