Pre-trained Projection Networks for NLP Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing models face challenges in deploying pre-computed language representations due to high memory requirements, making them unsuitable for resource-constrained environments like mobile devices, and hinder generalization to unknown words.
Innovation Solution
A pre-trained projection network that dynamically generates intermediate representations using a sequence of projection layers and intermediate layers, eliminating the need for storing embedding matrices and allowing on-the-fly computation, which can be transferred across various NLP tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If pre-computed word embedding matrices are stored for fast lookup, then language representation speed is improved, but memory requirements increase proportionally with vocabulary size
Solution Approach 1:
The patent extracts only the essential projection functions from the full embedding matrix, storing compact projection matrices (e.g., 100x100) instead of complete embedding tables (e.g., 50,000x300). This extraction allows dynamic generation of word representations through projection operations on input embeddings, dramatically reducing memory footprint while maintaining representation quality.
Solution Approach 2:
The patent replaces the mechanical lookup operation in embedding tables with a computational projection process. Instead of directly retrieving pre-computed embeddings through index lookup, the system applies projection matrices to input embeddings to dynamically generate word representations, substituting a compute-intensive operation for a memory-intensive one.
2Quantity of substance
If vocabulary is pruned to reduce memory requirements, then memory usage decreases, but ability to generalize to unknown words is hindered
Solution Approach 1:
The patent creates a universal projection matrix that can handle any word in the vocabulary space, including out-of-vocabulary words. The projection matrix operates on input embeddings to generate representations for any word, making the system versatile and adaptable to unknown words without requiring them to be pre-stored in the embedding table.
Solution Approach 2:
The patent pre-computes and stores compact projection matrices that capture the essential structure of the embedding space. These pre-computed projection matrices enable the system to quickly generate representations for any word through projection operations, including words not seen during training, by leveraging the learned projection transformations.
3Reliability
If large embedding matrices are used for comprehensive coverage, then representation quality is improved, but deployment on resource-constrained devices becomes infeasible
Solution Approach 1:
The patent extracts the essential transformation capabilities from large embedding matrices into compact projection matrices. By storing only the projection matrices (e.g., 100x100) rather than full embedding tables (e.g., 50,000x300), the system maintains representation quality while reducing model size to fit within resource-constrained devices like mobile phones and edge computers.
Solution Approach 2:
The patent changes the parameters of the representation system by using projection matrices with fewer parameters compared to full embedding matrices. This parameter reduction transforms a large-scale model into a compact version that can be deployed on-device, while the projection operations maintain the essential semantic transformation capabilities needed for quality representations.
Data Source
AI summary
Systems and methods are provided to pre-train projection networks for use as transferable natural language representation generators. In particular, example pre-training schemes described herein enable learning of transferable deep neural projection representations over randomized locality sensitive hashing (LSH) projections, thereby surmounting the need to store any embedding matrices because the projections can be dynamically computed at inference time.