Shared Embedding Layers for Compact Recurrent Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural language models is time-consuming due to the need to estimate the softmax function at every time stamp, and existing methods fail to efficiently reduce model size for deployment on computing devices with limited memory and space.
Innovation Solution
The method involves parameter sharing in the input and output embedding layers of neural language models by dividing word embedding vectors into sub-vectors and mapping them into a smaller set of shared sub-vectors, reducing the number of parameters and maintaining performance through structured parameter sharing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the softmax function is estimated at every time stamp, then the language model achieves accurate probability translation, but the training time increases significantly
Solution Approach 1:
The patent segments the large vocabulary space into multiple clusters, where each cluster is represented by a cluster center. Instead of computing softmax over the entire vocabulary at every time stamp, the model performs softmax only over the clustered representations, significantly reducing computational complexity while maintaining probability translation accuracy through the clustering structure.
2Measurement precision
If the model size is increased to improve language modeling performance, then the model achieves better accuracy, but the memory requirements and deployment difficulty increase
Solution Approach 1:
The patent merges the embedding layers by sharing parameters between the input embedding and output embedding matrices. This parameter sharing reduces the total number of parameters in the model while maintaining the ability to perform accurate language modeling, as the shared parameters capture common linguistic patterns that benefit both encoding and decoding processes.
Solution Approach 2:
The shared embedding parameters serve multiple functions simultaneously - they act as both input word embeddings and output word embeddings. This multi-functionality allows the model to achieve good language modeling performance with fewer parameters, as the same parameter set is utilized for both encoding input sequences and decoding output sequences.
3Quantity of substance
If the embedding layer parameters are reduced to decrease memory usage, then the model becomes more deployable, but the representation capability may be compromised
Solution Approach 1:
The patent introduces a clustering dimension to organize the embedding space. By projecting words into cluster centers and utilizing the cluster structure, the model achieves efficient representation with reduced parameters. The clustering approach creates a hierarchical organization that preserves semantic relationships while reducing the dimensional complexity of the embedding layer.
Data Source
AI summary
Described herein are systems and methods for compressing or otherwise reducing the memory requirements for storing and computing the model parameters in recurrent neural language models. Embodiments include space compression methodologies that share the structured parameters at the input embedding layer, the output embedding layers, or both of a recurrent neural language model to significantly reduce the size of model parameters, but still compactly represent the original input and output embedding layers. Embodiments of the methodology are easy to implement and tune. Experiments on several data sets show that embodiments achieved similar perplexity and BLEU score results while only using a fraction of the parameters.