AI Processor Data Prefetching for Matrix Memory Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence processors face challenges in efficiently prefetching data from off-chip memory to on-chip memory, particularly for matrix operations, leading to suboptimal performance and increased computational time.
Innovation Solution
A prefetching system for AI processors that includes a prefetcher with control and status registers, a matrix operation discrimination unit, and a request signal generation unit to determine and manage data prefetching based on matrix operation instructions, optimizing data movement between off-chip and on-chip memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is moved from off-chip memory to on-chip memory during computation, then data availability is maintained, but computational time increases due to sequential data movement operations
Solution Approach 1:
The patent implements a prefetcher that performs preliminary data movement operations by identifying future data requirements through matrix operation instructions and loading data into on-chip memory before it is actually needed for computation. This preliminary action overlaps data movement with computation time, eliminating sequential waiting periods and reducing total computational time while maintaining data availability.
2Productivity
If on-chip memory capacity is increased to store more data, then data reusability improves, but manufacturing cost and chip size increase
Solution Approach 1:
The prefetcher performs preliminary actions by predicting and loading only the specific data blocks required for upcoming matrix operations into on-chip memory. This selective prefetching approach maximizes data reusability by ensuring needed data is available in fast memory without requiring a large increase in on-chip memory capacity, thus avoiding proportional increases in chip size and manufacturing cost.
Solution Approach 2:
The system changes the parameter of data loading timing from synchronous (after computation is scheduled) to asynchronous (during computation execution). The prefetcher operates independently with its own control logic, allowing data to be loaded in the background while the processor executes computation instructions, effectively changing the temporal parameter of data availability without altering memory architecture.
3Productivity
If a dedicated prefetcher is implemented for matrix operations, then data movement efficiency improves, but device complexity increases
Solution Approach 1:
The prefetcher is designed with multi-functionality to handle both normal operations and matrix operations through a unified architecture. It can identify and process different operation types using the same basic prefetching mechanism, reducing the need for separate dedicated hardware for each operation type and thereby limiting the increase in device complexity while maintaining high data movement efficiency.
Data Source
AI summary
Disclosed herein are a prefetching device and method for an artificial intelligence processor. The prefetching method includes prefetching data, stored in external off-chip memory, into internal on-chip memory in the artificial intelligence processor, and storing information including an address value and a total amount of matrix operation data in at least one control and status register, as a kernel program is executed, extracting a matrix operation instruction among instructions provided from an instruction cache of the off-chip memory, determining whether prefetching is enabled based on a result of extracting the matrix operation instruction, as prefetching is enabled, determining a number of blocks to be prefetched based on the information stored in the at least one control and status register, and determining a bus burst value corresponding to the determined number of blocks and transmitting the bus burst value as a data request signal through a bus interface.