Transformer Cache Memory for Low-Precision Numerics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural network models require significant computational resources and memory, leading to inefficiencies in processing large data sizes due to high latency memory operations and challenges with data quantization.
Innovation Solution
The transformer compute apparatus includes a crossbar device, input and output buffers, a compute device, cache memory, and a memory device, which processes matrix inputs by determining projection tokens, converting them to block floating point format, and performing matrix operations to accelerate transformer computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in on-chip memory for fast access, then processing speed is improved, but memory capacity is insufficient to store all required elements
Solution Approach 1:
The memory system is segmented into multiple cache devices with different cache regions (first, second, and third cache regions). Each region stores different types of data (matrix inputs, projection tokens, converted tokens) to optimize access patterns for specific operations, thereby improving processing speed while efficiently utilizing limited on-chip memory capacity.
Solution Approach 2:
The patent implements a hierarchical memory structure where cache devices are nested within the compute apparatus, with multiple cache regions nested within each cache device. This nested architecture allows frequently accessed data to be stored in faster cache regions while less frequently accessed data resides in larger capacity regions, resolving the speed-capacity tradeoff.
2Ease of operation
If data is frequently transferred from memory storage device, then data availability is improved, but processing becomes memory bound due to high latency
Solution Approach 1:
Data is pre-loaded into cache regions before it is needed for computation. The third cache region stores matrix inputs, while the first and second cache regions store projection tokens and converted tokens in advance, allowing the compute device to access data without waiting for high-latency memory transfers during critical computation paths.
Solution Approach 2:
Cache devices act as intermediary storage between the slow memory storage device and the fast compute device. The crossbar converter device further mediates data format conversions, translating between floating point and block floating point formats to optimize both data availability and access speed for different computational stages.
3Productivity
If data is quantized into block floating point format, then computational efficiency is improved, but redundant quantizations and accuracy reduction occur
Solution Approach 1:
Different cache regions store data in different numerical formats optimized for their specific purpose. The first cache region stores floating point projection tokens for accurate representation, while the second cache region stores block floating point converted tokens for efficient computation. This local quality differentiation allows the system to achieve both accuracy and computational efficiency without redundant quantizations.
Data Source
AI summary
A transformer compute apparatus and method of operation therefor. The apparatus receives matrix inputs in a first format and generates projection tokens from these inputs. Among others, the apparatus includes a first cache device configured for processing first projection tokens and a second cache device configured for processing second projection tokens. The first cache device stores the first projection tokens in a first cache region and stores these tokens converted to a second format in a second cache region. The second cache device stores the second projection tokens converted to the second format in a first cache region and stores the converted second projection tokens after being transposed. Then, a compute device performs various matrix computations with the converted first projection tokens and transposed second projection tokens. Re-processing data and expensive padding and de-padding operations for transposed storage and byte alignment can be avoided using this caching process.