Matrix Data Conversion for Parallel Multi-Format AI Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based neural network models require significant computational resources and memory, making them inefficient and time-intensive, especially with the rapid growth of model sizes and compute requirements in natural language processing (NLP) applications.
Innovation Solution
The development of a matrix compute apparatus that can process matrix data in multiple formats by segmenting it into portions and using parallel processing within AI accelerator apparatuses and chiplet devices, allowing for efficient handling of different data formats without requiring separate hardware pathways, utilizing techniques like block floating point arithmetic and column blocking converters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If transformer models use self-attention mechanism for parallelism, then inference performance is improved, but computational intensity and memory requirements increase significantly
Solution Approach 1:
The patent segments matrix data into multiple data portions and processes them through multiple compute units in parallel. Each compute unit handles a specific portion of the matrix multiplication, allowing the system to maintain high computational intensity while distributing the workload across multiple units, thus improving overall inference performance without concentrating excessive computational demand in a single unit.
2Adaptability or versatility
If NLP models grow in size to increase capabilities, then model performance is improved, but training time and resource requirements increase dramatically
Solution Approach 1:
The patent divides large matrix operations into smaller segments that can be processed in parallel by multiple compute units. This segmentation allows larger models to be trained more efficiently by distributing the computational workload, thereby reducing training time while maintaining the ability to handle increased model sizes and capabilities.
Solution Approach 2:
The patent introduces a parallel processing dimension by utilizing multiple compute units that operate simultaneously on different data portions. This dimensional expansion from sequential to parallel processing enables the system to handle larger models without proportionally increasing training time, as multiple operations occur concurrently across different compute units.
3Adaptability or versatility
If matrix compute apparatus processes multiple data formats, then versatility is improved, but device complexity increases
Solution Approach 1:
The patent implements compute units that can process multiple data formats (e.g., different precision formats like FP32, FP16, INT8) through a unified computational framework. Each compute unit is designed with universal functionality to handle various data formats without requiring separate dedicated hardware pathways for each format, thus improving versatility while controlling device complexity through shared resources.
Data Source
AI summary
A method and device for data conversion in a matrix compute apparatus. The apparatus includes an input buffer (IB) that receives one or more matrix inputs characterized by a first format and a compute device coupled to the IB device that is configured to determine a combined matrix output. The compute device determines a first matrix output using a first input portion of the matrix input and determines a second matrix output using a second input portion. The compute device then determines a combined matrix output in a second format using the first and second matrix outputs. Within the compute device, an alignment device can determine a rounded matrix output from the combined matrix output, and a partial products reduction (PPR) device can determine a reduced matrix output in a third format using the rounded matrix output, which is stored in an output buffer (OB) coupled to the compute device.