Hyperformer Subvector Embeddings for Faster Transformer Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transformer models, such as RNNs and LSTMs, require extensive back-propagation training due to their deep network structure, leading to high resource demand and energy consumption, while transformers with fewer layers still require significant parameter counts, necessitating improvements to reduce resource usage without compromising performance.
Innovation Solution
Implementing a hyperformer with hyperspace embeddings that divide embedding vectors into subvectors, training them toward orthogonality using a low-rank hyperspace (LoRH) function, reducing parameter count and enhancing efficiency through compact model design.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformer models use deep network structures with many layers, then they can process and generate sequential data effectively, but back-propagation training takes a long time and consumes high resources
Solution Approach 1:
The patent segments the embedding vector into multiple subvectors (e.g., 6 subvectors of 8 dimensions each instead of one 48-dimensional vector). This segmentation allows the model to maintain deep network structure for effective processing while reducing the computational burden per layer through more efficient parameter utilization
Solution Approach 2:
The patent changes the parameter representation from traditional embedding vectors to hyperspace embeddings with subvectors that are trained toward orthogonality. This parameter transformation enables faster training by reducing the effective parameter count while maintaining model performance through the low-rank hyperspace projection
2Loss of time
If transformer models reduce the number of layers, then training time decreases, but the model depth is reduced which may affect processing capability
Solution Approach 1:
The patent introduces a new dimension of representation by transforming embedding vectors into hyperspace embeddings with multiple subvectors. This dimensional transformation allows the model to achieve effective processing with fewer layers by encoding information across multiple orthogonal subvector dimensions rather than relying solely on deep network stacking
3Device complexity
If transformer models use traditional embedding vectors, then the model structure remains simple, but information preservation is insufficient and resource consumption is high
Solution Approach 1:
The patent segments the embedding vector into multiple subvectors that are trained to be orthogonal to each other. This segmentation improves information preservation by distributing semantic information across independent subvector dimensions, reducing redundancy and enhancing the model's ability to capture nuanced relationships
Solution Approach 2:
The patent creates a composite embedding structure combining multiple subvectors with orthogonal relationships. This composite hyperspace embedding approach enhances information preservation by integrating multiple independent dimensional representations while maintaining computational efficiency through the low-rank projection
Data Source
AI summary
A plurality of tokens in an input sequence is rearranged in accordance with a number of the tokens in the input sequence, a number of subvectors per token, and a number of entries per subvector: an embedding vector for each token in the input sequence is generated and is divided into a plurality of subvectors to produce a hyperspace embedding. A positional encoding is added to the hyperspace embedding. The positionally encoded hyperspace embedding is processed in a decoder subnetwork: the positionally encoded hyperspace embedding is unfolded into a QKV representation, with a single query Q being obtained from the plurality of subvectors; the single query Q is applied to each subvector of the plurality of subvectors; and an attention function is calculated for each subvector. A plurality of output tokens is selected based at least in part on a result of the processing.