Relative Position-Aware Attention With Fast-Memory Tiling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicles face computational challenges in predicting the behaviors and trajectories of multiple objects due to the high computational expense and resource requirements of large machine-learned models, which can lead to latency issues and infeasible computations within the allowable reaction time.
Innovation Solution
Implementing an I/O-aware attention algorithm that utilizes fast memory, such as SRAM on GPUs, to efficiently compute relative position-aware attention scores by tiling matrices and incorporating relative position embeddings, reducing computational time and resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If larger transformer-based ML models are used to improve prediction accuracy, then prediction accuracy is improved, but computational expense and latency increase
Solution Approach 1:
The patent segments the computation process by separating position embedding calculations from the main transformer attention mechanism. Relative position embeddings are pre-computed and stored in lookup tables, allowing the main model to efficiently access position information without performing expensive real-time calculations. This segmentation enables larger models to run faster by dividing the computational workload into pre-processing and inference stages.
Solution Approach 2:
The patent performs preliminary action by pre-computing relative position embeddings before the main inference process. These pre-computed embeddings are stored in lookup tables that can be quickly accessed during attention calculations. By preparing position information in advance, the system eliminates the need for complex real-time position calculations, significantly reducing inference latency while maintaining model accuracy.
2Measurement precision
If larger transformer-based ML models are used to improve prediction accuracy, then prediction accuracy is improved, but computational resources and storage requirements increase
Solution Approach 1:
The patent uses copying by creating lookup tables that store pre-computed relative position embeddings. Instead of recalculating position information during each inference, the system copies and stores these embeddings in accessible memory structures. This copying approach allows larger transformer models to efficiently access position information without requiring proportional increases in computational resources, as the lookup tables provide constant-time access to position data.
3Device complexity
If standard attention mechanisms are used in transformer models, then model simplicity is maintained, but computational efficiency decreases
Solution Approach 1:
The patent introduces an intermediary mechanism by inserting lookup tables between the query-key computations and the attention score calculation. These lookup tables serve as mediators that provide pre-computed relative position embeddings to the attention mechanism. This intermediary approach maintains the simplicity of the standard transformer architecture while dramatically improving computational efficiency, as the lookup tables enable fast access to position information without requiring complex modifications to the attention mechanism itself.
Data Source
AI summary
A transformer-based machine-learned model may use relative positions between token embeddings to more accurately predict outputs. Example outputs may include an object detection, sensor data segmentation, object state prediction, and/or the like. Efficiently computing relative position-aware attention scores of the transformer-based machine-learned model may comprise utilizing fast memory available in specialized hardware according to the memory loading and flushing techniques described herein.