AI Processing Circuit Tensor Splitting for Memory Bandwidth Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial intelligence (AI) models in system-on-chip (SoC) systems face performance degradation due to high memory bandwidth requirements, which limit other modules' access to memory, necessitating a reduction in bandwidth usage.
Innovation Solution
A processing circuit and computation scheduling method that splits tensors and operators into sub-tensors and sub-operators, determines dependency relationships, and sorts them to optimize memory usage and bandwidth by managing the retention of target data in memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the AI model processes a large amount of data using high memory bandwidth, then the computation speed and processing capability are improved, but the memory bandwidth consumption increases, preventing other modules from accessing memory and decreasing overall system performance
Solution Approach 1:
The patent divides the computation process into multiple stages (first type operations and second type operations) and processes different sub-tensors in sequence. The memory management circuit reads tensors from external memory in segments, stores them in internal memory, and performs operations on sub-tensors separately. This segmentation allows the system to process data in manageable chunks, reducing the peak memory bandwidth requirement while maintaining computation throughput.
Solution Approach 2:
The memory management circuit performs preliminary actions by reading tensors from external memory and storing them in internal memory before the main computation begins. This pre-loading strategy prepares data in advance, allowing the computation unit to process sub-tensors without continuously accessing external memory, thereby reducing the overall memory bandwidth consumption during the computation phase.
2Reliability
If all intermediate data are retained in memory during computation, then the computation accuracy and completeness are maintained, but the memory usage increases, leading to higher memory costs and bandwidth requirements
Solution Approach 1:
The patent implements a strategy of discarding intermediate data from internal memory after it has been used for computation. The memory management circuit deletes intermediate data from the memory based on the operation order and dependency relationships. This allows the system to reuse memory space for new data, significantly reducing peak memory usage while maintaining computation accuracy through proper data management.
Solution Approach 2:
The system dynamically manages memory resources by adjusting which data are retained and which are deleted based on the current computation stage and dependency relationships. The memory management circuit determines the operation order and selectively retains only the necessary intermediate data required for subsequent operations, optimizing the balance between memory usage and computation accuracy.
3Quantity of substance
If the computation process is divided into multiple stages with intermediate data retention, then the system can optimize memory bandwidth usage, but the device complexity increases due to the need for memory management circuits and operation scheduling
Solution Approach 1:
The memory management circuit performs multiple functions: it manages data transfer between external and internal memory, tracks intermediate data, determines operation order based on dependency relationships, and deletes intermediate data when no longer needed. By consolidating these diverse functions into a single multi-functional circuit, the patent reduces the overall device complexity compared to having separate dedicated circuits for each function.
Solution Approach 2:
The patent combines the computation unit and memory management circuit into an integrated processing system. The memory management circuit is directly coupled to both the external memory and the computation unit, allowing it to coordinate data flow and memory operations efficiently. This merging of functions into a unified architecture reduces the number of separate components and interconnections, thereby reducing device complexity.
Data Source
AI summary
A processing circuit of an artificial intelligence (AI) model includes a memory, a memory management circuit, and an operation circuit. The memory management circuit reads a tensor from an external memory and stores the tensor in the memory. The operation circuit is configured to perform the following operations: performing an operation of a first type on a first and second sub-tensors of the tensor to generate a first and second intermediate data, respectively; performing an operation of a second type on the first intermediate data and the second intermediate data to generate a third intermediate data; performing an operation of the first type on a third sub-tensor of the tensor to generate a fourth intermediate data; and performing an operation of the second type on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data.