AI Model Operator Segmentation for Memory Bandwidth Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI models are inefficient in utilizing internal memory and result in increased bandwidth requirements due to processing one batch of input data at a time, leading to wasted hardware resources and reduced performance.
Innovation Solution
A method for creating and executing AI models that involves dividing operators based on batch thresholds, intermediate tensor life cycles, sizes, and memory capacity, allowing for simultaneous processing of multiple batches and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the processing unit processes only one batch of input data at a time, then the internal memory cannot be fully utilized, but the hardware resources are wasted
Solution Approach 1:
The patent divides the processing of multiple batches into segmented stages, where the AI model is split into multiple operator groups that can process different batches simultaneously at different stages. This segmentation allows the internal memory to be fully utilized across multiple batches while maintaining high processing throughput through parallel execution.
Solution Approach 2:
The patent implements dynamic batch processing where the number of batches processed simultaneously (N) can be adjusted based on memory capacity and bandwidth requirements. The system dynamically optimizes the balance between memory utilization and processing throughput by adjusting the batch parallelism degree during execution.
2Adaptability or versatility
If the internal memory cannot be fully utilized, then the processing unit needs to frequently read data from external memory, but the bandwidth requirement increases
Solution Approach 1:
The patent performs preliminary actions by pre-loading necessary data into the internal memory before processing begins. By calculating the required memory capacity based on the batch threshold and intermediate tensor sizes, the system prepares the internal memory in advance to hold all necessary data for parallel batch processing, thereby reducing frequent external memory accesses and lowering bandwidth requirements.
Solution Approach 2:
The patent introduces the concept of intermediate tensors as mediators between different operator groups and batches. These intermediate tensors are managed within the internal memory, acting as an intermediary buffer that allows data to be shared and reused across multiple batches without requiring frequent external memory accesses, thus reducing bandwidth requirements.
3Productivity
If the internal memory cannot be fully utilized, then the processing efficiency decreases, but the circuit system performance is reduced
Solution Approach 1:
The patent ensures continuity of useful action by maintaining full internal memory utilization throughout the processing of multiple batches. By organizing operators into groups that continuously process different batches in parallel, the system eliminates idle memory periods and maintains continuous productive work, thereby improving both processing efficiency and circuit system performance simultaneously.
Data Source
AI summary
A method for creating an artificial intelligence (AI) model is applied to an intelligence processing unit (IPU). The IPU includes a computing circuit and a memory. The AI model includes a plurality of operators. The computing circuit generates an intermediate tensor in the process of executing each operator. The method includes the following steps: (A) dividing the operators according to a batch threshold, life cycles of the intermediate tensors, sizes of the intermediate tensors, and a capacity of the memory; (B) calculating a bandwidth requirement of the IPU for an external memory when executing the AI model; and (C) storing a relationship between the batch threshold and the bandwidth requirement. The AI model performs an operation of the same operator on N batches of input data substantially at the same time, and N is a positive integer.