Matrix Kernel Leading Dimension Control for Cache Conflict Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning workloads face performance issues due to cache conflicts caused by incompatible leading dimensions in matrix multiplication and convolution operations, leading to evictions from the cache and reduced performance.
Innovation Solution
The solution involves controlling the dimension size of matrices to prevent cache conflicts by limiting the leading dimension to be less than or equal to the number of ways in the set-associative cache, and introducing additional reuse dimensions to optimize the ratio of floating-point instructions to memory read instructions, while using inline copies and bilateral buffer reuse to enhance performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If matrix data is stored to cache memory for rapid retrieval, then access speed is improved, but cache conflicts occur when leading dimensions are incompatible with cache layout, causing data eviction and performance degradation
Solution Approach 1:
The patent changes the leading dimension parameter of the matrix kernel to be compatible with the cache layout. Specifically, it transforms an incompatible leading dimension (e.g., 1024) into a compatible one (e.g., 512) by adjusting the kernel implementation, thereby eliminating cache conflicts and improving the cache hit rate while maintaining fast data access
Solution Approach 2:
The patent dynamically selects different kernel implementations based on the compatibility between the leading dimension and cache layout. When incompatibility is detected, it automatically switches to a transformed kernel with adjusted leading dimensions, making the system adaptive to different matrix sizes and cache configurations
2Reliability
If the leading dimension is limited to be less than or equal to the number of cache ways, then cache conflicts are reduced, but the flexibility in handling arbitrary matrix sizes is constrained
Solution Approach 1:
The patent segments the matrix multiplication task into multiple smaller operations that can be handled by kernels with leading dimensions compatible with the cache layout. By dividing a large matrix operation into smaller sub-operations, each with adjusted leading dimensions, it maintains cache efficiency while supporting arbitrary input matrix sizes
Solution Approach 2:
The patent introduces an intermediary transformation step that converts arbitrary matrix dimensions into forms compatible with cache requirements. This intermediary process adjusts the leading dimension parameter before kernel execution, acting as a mediator between user-defined matrix sizes and cache optimization requirements
3Productivity
If additional reuse dimensions are introduced to optimize the ratio of floating-point instructions to memory read instructions, then computational efficiency is improved, but the complexity of kernel implementation increases
Solution Approach 1:
The patent introduces additional reuse dimensions (e.g., transforming a 2D matrix operation into a 3D operation with an added dimension) to increase the ratio of computing operations to memory accesses. This dimensional transformation enables better data reuse and computational efficiency while managing complexity through systematic dimension management
Data Source
AI summary
Systems, apparatuses and methods may provide for technology that determines a ratio of floating point instructions to memory read instructions and controls a dimension size of a matrix kernel based at least in part on the ratio. In one example, the matrix kernel conducts an operation between a first matrix and a second matrix and the technology reuses elements of the first matrix for multiple vector lines of the second matrix.