Matrix Accelerator Load Store Cache for Tensor Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current programming models for graphics processors require extensive address calculations for data access, reducing the efficiency of matrix/tensor operations due to the need for manual management of large data transfers, which hampers performance in compute workloads.
Innovation Solution
The implementation of hardware enhancements for batch memory access instructions that allow efficient loading and storing of 2D blocks from n-dimensional tensor data, offloading scalar address calculations to a dedicated Load Store Cache unit, thereby reducing latency and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual address calculations are used for data access in existing programming models, then programmers can control data transfer at lower levels of abstraction, but the overall efficiency of computations is reduced due to extensive address calculations and manual management of large data transfers
Solution Approach 1:
The patent introduces hardware intermediaries (Load Store Cache unit and batched memory access instructions) that mediate between the software kernel and memory access operations. These hardware components automatically handle address calculations and data transfer management, freeing the software from manual control while maintaining high computational efficiency.
Solution Approach 2:
The patent replaces manual mechanical address calculations with automated hardware-based memory access mechanisms. The Load Store Cache unit and batched memory access instructions substitute the software-based address calculation process with hardware-accelerated operations, significantly improving computational efficiency while reducing the burden on software kernels.
2Reliability
If extensive address calculations are performed for memory access, then correct data can be accessed in memory, but latency increases and performance is reduced
Solution Approach 1:
The patent implements preliminary action by pre-computing and storing batched memory access patterns in the Load Store Cache unit before actual data access occurs. This allows the hardware to automatically retrieve correct data without performing time-consuming address calculations during the critical data access phase, thereby reducing latency while maintaining correctness.
Solution Approach 2:
The Load Store Cache unit acts as an intermediary that pre-processes address calculations and stores them in advance. This intermediary layer absorbs the computational overhead of address calculations away from the main computation timeline, ensuring both correct data access and reduced latency.
3Productivity
If manual management of large data transfers is performed by software kernels, then data can be transferred between memory and processor, but the computational burden on software increases
Solution Approach 1:
The patent introduces the Load Store Cache unit as a hardware intermediary that handles data transfer operations between memory and processor. This intermediary absorbs the complexity of data transfer management, including address calculations and data movement coordination, thereby reducing the computational burden on software kernels while maintaining high data transfer capability.
Solution Approach 2:
The patent replaces software-based data transfer management with hardware-accelerated mechanisms. The batched memory access instructions and Load Store Cache unit substitute the mechanical process of manual data transfer management with automated hardware operations, significantly reducing software complexity while preserving data transfer productivity.
Data Source
AI summary
Embodiments described herein provide a system to enable access to an n-dimensional tensor in memory of a graphics processor via a batch of two-dimensional block access messages. One embodiment provides a graphics processor comprising general-purpose graphics execution resources coupled with the system interface, the general-purpose graphics execution resources including a matrix accelerator. The matrix accelerator is configured to perform a matrix operation on a plurality of tensors stored in a memory. Circuitry is included to facilitate access to the memory by the general-purpose graphics execution resources. The circuitry is configured to receive a request to access a tensor of the plurality of tensors and generate a batch of two-dimensional block access messages along a dimension of n>2 of the tensor. The batch of two-dimensional block access messages enables access to the tensor by the matrix accelerator.