Matrix Operation Acceleration Through Preconverted Memory Layouts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing matrix multiplication accelerators face performance bottlenecks due to complex data movement and layout conversion requirements between host and accelerator memories, particularly when using different layouts, leading to inefficient computing efficiency.
Innovation Solution
The method involves moving slices of matrices between host and accelerator memories with the same layout, allowing for simplified data movement instructions and eliminating the need for on-the-fly layout conversion, thereby reducing time consumption and improving computing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If on-the-fly layout conversion is performed during data transmission using DMA, then the computing speed is improved, but the control process becomes complex and requires complex software loops
Solution Approach 1:
The patent applies preliminary action by pre-converting the layout of the result matrix in the host memory to match the accelerator memory layout before data transmission. This eliminates the need for complex on-the-fly layout conversion during DMA operations, simplifying the control process while maintaining high computing speed.
2Productivity
If different layouts are used in host memory and accelerator memory, then data transmission can be optimized, but the time consumed by data movement operations increases
Solution Approach 1:
The patent converts the result matrix layout in host memory to match the accelerator memory layout before transmission. This preliminary conversion reduces the time consumed during data movement operations by eliminating the need for complex on-the-fly layout conversion, thereby improving overall computing efficiency.
3Adaptability or versatility
If complex data movement instructions are used to achieve on-the-fly layout conversion, then layout flexibility is improved, but the computing efficiency decreases
Solution Approach 1:
The patent performs layout conversion of the result matrix in advance in the host memory before data transmission to the accelerator. This approach maintains layout flexibility while improving computing efficiency by avoiding complex on-the-fly layout conversion instructions during data movement operations.
Data Source
AI summary
An acceleration method for performing matrix operations by a computing device, includes: moving a first slice of a left matrix and a second slice of a right matrix from a host memory to an accelerator memory; moving a third slice of a result matrix from the host memory to the accelerator memory; performing a matrix operation on the first and second slices to obtain a fourth slice of the result matrix; performing a vector operation on the third and fourth slices to obtain a fifth slice of the result matrix; and moving the fifth slice from the accelerator memory to the host memory. The first, second, and fourth slices on the accelerator memory are respectively in a first 4-dimensional (4D) layout, second 4D layout, and third 4D layout. The third slice and the fifth slice are in the same layout on the host memory and the accelerator memory.