Matrix Processing Architecture for Parallel Compute Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hardware accelerators for matrix multiplication face challenges in optimizing memory bandwidth when parallel processing is employed, leading to compute unit starvation or underutilization due to mismatched data storage formats and increased memory reads, particularly when processing large matrices.
Innovation Solution
A processor architecture with multiple processing elements connected via bidirectional point-to-point links and shared memory, allowing for efficient data exchange and transposition of matrix segments, enabling parallel processing without redundant memory accesses and maintaining efficient memory bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple processors are used in parallel to process matrix multiplication, then computational power increases, but memory bandwidth consumption increases proportionally more, leading to compute unit starvation
Solution Approach 1:
The patent divides the matrix data into segments distributed across multiple processing elements. Each processing element holds a portion of the matrix data locally, and through point-to-point links, adjacent processing elements exchange only the specific segments they need. This segmentation allows parallel processing while reducing redundant memory accesses, as each processor works with its local segment and exchanges only necessary data with neighbors.
Solution Approach 2:
The patent introduces a spatial dimension for data exchange by organizing processing elements in a grid layout with point-to-point links between adjacent elements. Instead of all processors competing for the same shared memory bandwidth, data flows through multiple spatial paths (horizontal and vertical links), distributing the memory bandwidth demand across different dimensional channels and reducing contention.
2Adaptability or versatility
If data is stored in shared memory for parallel access, then accessibility improves, but data format mismatch between storage and compute units introduces latency and reduces efficiency
Solution Approach 1:
The patent performs preliminary organization of data in shared memory according to the specific access patterns required by the parallel processing algorithm. Matrix data is stored in a format that facilitates efficient distribution to processing elements and predictable exchange patterns between adjacent elements. This preliminary arrangement eliminates the need for runtime data reordering and format conversion, reducing latency while maintaining accessibility.
3Ease of operation
If processing elements exchange data frequently through shared memory, then data availability improves, but memory read requirements increase, reducing overall efficiency
Solution Approach 1:
The patent introduces point-to-point communication links as intermediary channels between adjacent processing elements. Instead of all processing elements accessing data through the shared memory bottleneck, adjacent elements can exchange data directly through dedicated links. This intermediary communication path reduces the number of memory reads required, as data is exchanged locally rather than being repeatedly fetched from shared memory, thereby improving processing efficiency while maintaining data availability.
Data Source
Figure 1~2
Figure 3A~3C
Figure 4
AI summary
A method is disclosed for block processing two matrices stored in a same shared memory, one ([a]) being stored by rows and the other ([b]) being stored by columns, using a plurality of processing elements (PE), where each processing element (PE0) is connected to the shared memory by a respective N-bit access and to a first adjacent processing element (PE1) by a bidirectional N-bit point-to-point link. The method comprising the following steps carried out in one processor instruction cycle: receiving (LV) in the processing elements (PE0-PE3) respective different N-bit segments of a same one of the two matrices ([b]) by the respective memory accesses; and exchanging (SEND.PE1, RECV.PE1) with the first adjacent processing element (PE1), by means of the point-to-point link, N-bit segments of a first ([a]) of the two matrices which were received in the adjacent processing elements (PE0, PE1) in a previous instruction cycle.