Processor ABI for Thread Block Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems, particularly GPUs, face limitations in achieving high bandwidth and low latency performance due to the need to propagate data through hierarchical memory systems, which hampers applications like deep learning that require efficient data propagation between operators.
Innovation Solution
The implementation of an application binary interface (ABI) that allows results from one operator to be stored in registers and shared memory, enabling subsequent operators to consume them without reading from global memory, thereby optimizing data propagation between thread blocks and reducing reliance on global memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is propagated through hierarchical memory system, then data storage capacity is improved, but bandwidth and latency performance deteriorates
Solution Approach 1:
The patent introduces shared memory as an intermediary between global memory and registers. Results from one operator are stored in shared memory, which acts as a mediator that enables subsequent operators to consume data without accessing global memory, thus resolving the contradiction between storage capacity and access performance
Solution Approach 2:
The patent creates a new data access dimension by implementing thread block-to-thread block communication through shared memory. This additional communication path allows operators to exchange data locally without traversing the traditional global memory hierarchy, effectively adding another dimension to the memory access model
2Adaptability or versatility
If global memory access is used for operator communication, then data accessibility is improved, but power consumption and resource usage increases
Solution Approach 1:
The patent applies local quality by making shared memory accessible to specific thread blocks that need to communicate, rather than making all memory uniformly accessible. This localized access model reduces unnecessary memory transactions and associated power consumption while maintaining data accessibility for relevant operators
3Adaptability or versatility
If hierarchical memory propagation is used, then system compatibility is maintained, but deep learning framework efficiency deteriorates
Solution Approach 1:
The patent segments the memory access model into distinct levels: global memory for general storage, shared memory for operator-to-operator communication, and registers for intra-operator data. This segmentation allows deep learning frameworks to optimize frequently accessed data in faster memory segments while maintaining compatibility with existing global memory access patterns
Data Source
AI summary
An application binary interface (ABI) can be exposed in a processor to enable blocks of threads, which may correspond to separately compiled operators, to communicate without storing data to global memory external to the processor. The ABI can define how results of one computation, corresponding to a first thread block, will be organized in registers and shared memory of a processor at the end of one operator (i.e., kernel). The start of the next operator (i.e., kernel), corresponding to a second thread block, can consume the results from the registers and shared memory. Data can be stored to processor local storage for individual threads as they exit the block. Once published, libraries can be separately compiled, optimized, and tested as long as they adhere to the published ABI.