Cache-Aware Matrix Multiplication for Lower Memory Access Costs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning models with memory-intensive operators, such as General Matrix Multiplication (GEMM), face challenges in efficiently utilizing cache memory, leading to increased memory access costs and underutilization of storage space.
Innovation Solution
The apparatus and method involve decomposing input matrices into sub-matrices and determining optimal storage and access strategies to minimize memory access costs by fully utilizing cache resources, allowing for parallel matrix multiplication operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If input matrices are processed as whole units in cache memory, then storage simplicity is maintained, but memory access costs increase and cache utilization decreases
Solution Approach 1:
The patent divides input matrices into multiple sub-matrices that can be processed independently in cache memory. This segmentation allows the system to load and process smaller matrix blocks that fit within cache capacity, reducing memory access costs and improving cache utilization while maintaining manageable complexity through systematic decomposition patterns
2Power
If cache memory is allocated for large matrix operations, then computational capability is improved, but cache space utilization decreases leading to wasted resources
Solution Approach 1:
By segmenting large matrix operations into smaller sub-matrix operations, the patent enables efficient utilization of limited cache space. Each sub-matrix operation uses only the necessary cache resources, avoiding allocation of excessive cache memory while maintaining high computational throughput through pipelined processing of multiple sub-matrices
Solution Approach 2:
The patent processes matrices in partial blocks that fit within cache capacity, performing multiple passes over the data. This partial action approach ensures that cache space is fully utilized for each operation rather than allocating excessive cache for entire matrices, optimizing the balance between computational power and resource utilization
Data Source
Figure 1~2A
Figure 2B~2C
Figure 3
AI summary
The present disclosure provides an apparatus of processing data, which relates to a field of artificial intelligence technology, and in particular to a field of chip technology. The apparatus includes: a cache unit including a plurality of storage spaces; a processor configured to: determine I groups of storage space from the plurality of storage spaces; perform an operation on each group of storage space to obtain a plurality of first initial memory access costs corresponding to the group of storage space, where the operation includes: determining a plurality of first initial shape information according to a shape of a first matrix and a capacity of the first storage space; determining at least one second shape information according to each of the plurality of first initial shape information, where the second shape information is related to a second matrix; and determining the plurality of first initial memory access costs according to a plurality of second shape information and the plurality of first initial shape information; and determine a target memory access cost from all first initial memory access costs of the I groups of storage space. The present disclosure further provides a method of processing data, an electronic device, and a storage medium.