AI Model Partition Processing With Streamlined Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators are inadequate for processing hyper-scale AI models due to computational load and delay time issues, particularly in applications like natural language processing, and suffer from data bottlenecks and inefficiencies in parallelizing transformer-based generation models.
Innovation Solution
A latency processing unit (LPU) that optimizes computational load and delay time by maximizing external memory bandwidth through streamlined memory access (SMA) and a streamlined execution engine (SXE), utilizing a structure with MAC trees, high bandwidth memory, and a vector execution engine, along with an instruction scheduling unit to manage operations efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing hardware accelerators use data parallelism to maximize computational load, then computational capacity is improved, but delay time increases and data bottlenecks occur
Solution Approach 1:
The system segments the AI model into multiple partitions and distributes them across multiple LPUs. Each LPU handles a specific partition independently, enabling parallel processing while reducing inter-dependency delays. The segmentation of computational tasks across multiple independent units allows the system to maintain high computational load while minimizing delay time through concurrent execution.
Solution Approach 2:
The patent transitions from traditional data parallelism to a partition-based dimensional approach where the model is divided along the partition dimension rather than data dimension. This dimensional change allows simultaneous access to different model partitions without data bottleneck conflicts, achieving both high computational load and reduced delay time through multi-dimensional parallel processing.
2Measurement precision
If hyper-scale AI models increase model size for higher accuracy, then accuracy is improved, but data bottlenecks occur in existing hardware accelerators
Solution Approach 1:
Large-scale AI models are segmented into multiple manageable partitions that can be distributed across multiple LPUs. Each partition is processed independently by dedicated hardware units, eliminating the data bottleneck that occurs when trying to process entire large models through a single accelerator. This segmentation enables the system to handle hyper-scale models with high accuracy without encountering data transmission bottlenecks.
Solution Approach 2:
The system introduces partition management and distributed memory structures as intermediaries between the large AI model and the processing units. These intermediaries facilitate efficient data distribution and retrieval across multiple LPUs, preventing data bottlenecks while supporting the processing of large-scale high-accuracy models.
3Adaptability or versatility
If transformer-based generation models use attention operations, then model capability is improved, but parallelization becomes difficult
Solution Approach 1:
The attention operation computation is segmented into discrete computational stages that can be executed in parallel across different LPUs. By dividing the attention mechanism into independent computational blocks operating on different partitions, the system maintains the sophisticated model capability of transformer-based attention while enabling effective parallelization through spatial distribution of computational tasks.
4Productivity
If existing hardware accelerators maximize computational load, then processing capacity is improved, but memory bandwidth utilization is insufficient
Solution Approach 1:
The system performs preliminary actions by pre-loading and caching frequently accessed model partitions and data in high-speed memory buffers before they are needed for processing. This preliminary preparation ensures that when computational tasks are executed, the required data is already available in fast memory, maximizing memory bandwidth utilization and supporting high processing capacity without energy inefficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided is a latency processing unit. The latency processing unit may include a plurality of multiplier-accumulator (MAC) trees configured to perform a matrix product operation for at least one of a plurality of partitions that implement an artificial intelligence (AI) model, streamlined memory access configured to connect each of the plurality of MAC trees to high bandwidth memory in which the at least one partition has been stored through a plurality of channels, a vector execution engine configured to perform an additional operation on results of the operation of the plurality of MAC trees, a local memory unit configured to store the results of the operation of the vector execution engine and an activation value, and an instruction scheduling unit configured to schedule the operations of the plurality of MAC trees and the vector execution engine.