GPU ML Clusters With Adjacent Memory for Low-Latency Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU architectures face challenges in improving power efficiency of machine learning arithmetic units (ML ALUs) while managing the area cost and data delivery latency, which are dedicated to compute units (CUs) for matrix multiplication operations.
Innovation Solution
Implementing a graphics processing unit (GPU) with separate voltage domains for ML clusters, adjacent memory modules, and direct memory access engines to reduce data delivery costs and latency by using private buses and scratchpad memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If separate voltage domains are used for ML ALUs, then power efficiency is improved, but area cost increases
Solution Approach 1:
The patent merges ML ALUs with memory modules into integrated ML clusters, where multiple ML ALUs share common memory and control logic. This integration reduces the area overhead of separate voltage domains while maintaining power efficiency benefits through localized data access and reduced memory interface requirements.
Solution Approach 2:
The ML clusters are designed as multi-functional units that can perform both traditional GEMM operations and specialized ML workloads. By making the clusters universal, the patent reduces the need for dedicated separate structures for different voltage domains, as the same physical infrastructure serves multiple purposes across different operational modes.
2Productivity
If ML ALUs are dedicated to compute units, then matrix multiplication performance is improved, but data delivery latency increases
Solution Approach 1:
The patent implements a nested memory hierarchy where scratchpad memory is embedded within each ML cluster, which in turn is part of the larger GPU memory system. This nested structure allows data to be cached locally in scratchpad memory close to the ML ALUs, reducing the time to deliver data while maintaining the dedicated compute unit architecture for high-performance matrix operations.
Solution Approach 2:
The patent introduces scratchpad memory as an intermediary buffer between the main memory and ML ALUs. This intermediary layer pre-loads and caches frequently accessed data, mediating the data delivery process to reduce latency while allowing the dedicated ML ALUs to maintain their high computational throughput for matrix multiplication operations.
3Power
If ML ALUs consume significant GPU power, then computational capability is improved, but overall power efficiency deteriorates
Solution Approach 1:
The patent segments the GPU into multiple independent ML clusters, each with its own voltage domain and power management. This segmentation allows selective power gating of individual clusters based on workload requirements, enabling high computational capability in active clusters while maintaining overall power efficiency by powering down inactive clusters. Each cluster can be independently scaled and managed.
Data Source
AI summary
A graphics processing unit (GPU) for clustering of machine learning (ML) functional components, including: a plurality of compute units; a plurality of ML clusters, wherein each of the ML clusters comprises at least one arithmetic logic unit (ALU), and wherein each of the ML clusters is associated with a respective subset of the compute units; and a plurality of memory modules each positioned on the GPU adjacent to a respective ML cluster of the plurality of ML clusters, wherein each ML cluster is configured to directly access one or more adjacent memory modules.