Near-Memory Compute Node for Reduce-Scatter Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed neural network training is time-consuming and computation-intensive, with all-reduce operations demanding high interconnect and memory bandwidth, competing for resources with other operations like GEMM, and requiring significant computational burden on main processors, GPUs, and FPGAs.
Innovation Solution
Offloading distributed reduction operations, such as reduce-scatter operations, to near-memory computation units with PIM-enabled memory, reducing memory bandwidth demand and minimizing interference with concurrently executing kernels like GEMM by performing these operations closer to memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed reduction operations are performed on main processors or GPUs, then computation accuracy is maintained, but computational burden and memory bandwidth demand increase significantly
Solution Approach 1:
The patent extracts the reduction operation from the main processor/GPU and places it in a dedicated reduction unit. This separation allows the main processor to focus on other tasks while the reduction unit handles reduction operations independently, reducing computational burden on main processors while maintaining computation accuracy through dedicated hardware implementation.
Solution Approach 2:
The patent introduces a reduction unit as an intermediary component between memory and the main processor/GPU. This intermediary handles the computationally intensive reduction operations, reducing memory bandwidth demand by processing data closer to its source and preventing the main processor from becoming a bottleneck.
2Productivity
If all-reduce operations are executed with high bandwidth demand, then computation throughput is improved, but interference with concurrently executing kernels increases
Solution Approach 1:
The patent segments the computing system into distinct functional units: a main processor/GPU for executing computation kernels and a separate reduction unit for handling reduction operations. This segmentation allows both operations to proceed concurrently with minimal interference, as each unit operates independently with its own resource allocation.
Solution Approach 2:
The reduction unit acts as an intermediary that handles high-bandwidth reduction operations without blocking the main processor's access to memory. By placing the reduction unit between memory and the main processor, the system can maintain high computation throughput while minimizing interference with concurrently executing kernels.
3Loss of energy
If reduction operations are performed closer to memory, then memory bandwidth demand is reduced, but device complexity increases
Solution Approach 1:
The reduction unit is designed as a multi-functional component that can perform various reduction operations (sum, max, min, etc.) and work with different data types. This universal design reduces the need for multiple specialized units, thereby reducing overall device complexity while still achieving the benefit of reduced memory bandwidth demand through near-memory computation.
Data Source
AI summary
An apparatus for performing distributed reduction operations using near-memory computation includes memory and a first near-memory compute node. The first-near-memory compute node is coupled to a plurality of near-memory compute nodes. The first near-memory compute node comprises logic to store first data loaded from a second near-memory compute node, perform a reduction operation on the first data and second data to compute a result; and store the result within the first near-memory compute node. In some aspects, the near-memory compute node includes a PIM execution unit and carries out the reduction operation utilizing PIM commands.