AI Accelerator Reuse Data Management for Memory Access Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI hardware accelerators face challenges in efficiently managing memory access costs and optimizing reuse data to minimize external memory accesses during workload processing.
Innovation Solution
The proposed solution involves an operation method for an AI accelerator that determines reuse data based on hardware resource information and memory access costs, storing this data in a multilevel memory to reduce external memory accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If reuse data is determined based on hardware resource information and memory access cost, then external memory access is minimized, but computation overhead increases
Solution Approach 1:
The system performs preliminary analysis of hardware resource information and memory access costs before executing workloads to determine reuse data. This advance preparation allows the accelerator to identify which data should be retained in multilevel memory, avoiding costly external memory accesses during actual computation without adding overhead during runtime execution
Solution Approach 2:
The accelerator automatically determines reuse data by analyzing its own hardware resource information and memory access costs without requiring external intervention. The system self-optimizes by identifying patterns in its own operation, determining which intermediate values should be cached in multilevel memory based on its specific hardware characteristics and workload patterns
2Loss of energy
If multilevel memory is used to store reuse data, then external memory access cost is reduced, but device complexity increases
Solution Approach 1:
The memory system is segmented into multiple levels (L0, L1, L2) with different access characteristics and capacities. Each memory level serves a specific purpose: L0 for fast per-element storage, L1 for shared workspace, and L2 for larger reuse data storage. This segmentation allows the system to optimize for different types of memory access patterns without requiring a completely complex unified memory architecture
Solution Approach 2:
Different memory levels are assigned different qualities and characteristics appropriate to their function. L0 memory provides fastest access for individual processing elements, L1 provides medium-speed shared access for groups of elements, and L2 provides slower but larger-capacity storage for reuse data. Each memory level is optimized locally for its specific access patterns and performance requirements
3Productivity
If reuse data is dynamically determined based on workload characteristics, then processing efficiency is improved, but control complexity increases
Solution Approach 1:
The system implements feedback mechanisms where the accelerator continuously monitors hardware resource usage and memory access patterns during workload execution. Based on this feedback, the system dynamically adjusts which data should be classified as reuse data and stored in multilevel memory, optimizing performance for different workload characteristics without requiring complex manual configuration
Solution Approach 2:
The reuse data determination is made dynamic rather than static, allowing the system to adapt to different workload characteristics. The accelerator can identify patterns in different types of AI workloads (e.g., convolution operations, matrix multiplications) and automatically adjust which intermediate values should be retained in multilevel memory based on the specific characteristics of the current workload being executed
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An accelerator, an operation method of the accelerator, and an accelerator system including the accelerator are disclosed. The operation method includes receiving one or more workloads assigned by a host controller, determining reuse data of the workloads based on hardware resource information and/or a memory access cost of the accelerator when a plurality of processing units included in the accelerator performs the workloads, and providing a result of performing the workloads.