GPU Cache Bypass and Local Access for Lower Power Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GPU processing and caching architectures face issues with power, latency, and bandwidth inefficiencies, particularly in atomic reduction operations, shared caching scenarios, and last-level cache management, which impact performance scalability and battery life.
Innovation Solution
Implementing a streaming buffer between producer and consumer IP cores to bypass the mid-level cache, using a double buffering technique, and optimizing cache structures to reduce unnecessary cache access and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If atomic reduction operations are performed in shared local memory or L3 cache, then the operations can be executed, but bandwidth is consumed and latency is created
Solution Approach 1:
The patent segments the atomic reduction operation into two distinct phases: a read phase where multiple threads simultaneously read values from memory without contention, and a write phase where results are consolidated and written back. This segmentation eliminates the bandwidth consumption and latency associated with traditional locked atomic operations by removing the critical section bottleneck.
Solution Approach 2:
The patent performs preliminary reading of values from memory into thread-local registers before any processing occurs. By pre-fetching data into fast registers where threads have exclusive access, the system avoids subsequent memory contention and latency during the reduction computation, allowing all threads to operate independently and simultaneously.
2Productivity
If video tiles are written to DRAM for AI inference, then rendering is completed, but huge bandwidth and latency costs are incurred when reading back to GPU
Solution Approach 1:
The patent extracts the AI inference workload from the traditional rendering pipeline by allowing compute shaders to directly access and process tile data in VRAM without requiring a read-back to the host CPU. This extraction eliminates the expensive memory bandwidth consumption associated with transferring data between GPU and host memory, enabling energy-efficient AI processing directly on the GPU.
Solution Approach 2:
The patent makes the GPU memory system multi-functional by enabling it to serve both as render target memory and as input memory for AI inference operations. By allowing compute shaders to directly consume tile data from VRAM, the system eliminates the need for separate memory transfers and enables the same memory bandwidth to serve dual purposes: rendering and AI processing.
3Ease of operation
If a single IP core accesses memory through the central fabric to a mid-level cache, then data access is possible, but cache power and central fabric power are consumed impacting battery life
Solution Approach 1:
The patent merges the mid-level cache functionality directly into the IP core that needs it, creating a private cache hierarchy. By combining the cache and the processing unit into a single integrated block, the system eliminates the need for data to traverse the power-consuming central fabric interconnect, allowing the IP core to access its private cache through low-power on-core interconnects.
Solution Approach 2:
The patent introduces a private cache as an intermediary between the IP core and main memory, eliminating the need for the IP core to use the central fabric for data access. This intermediary cache sits locally within the IP core, providing fast data access without requiring communication through the power-hungry central fabric interconnect.
4Quantity of substance
If the last-level cache is made larger to improve caching, then cache capacity increases, but it gets further away from shader cores driving up power and latency
Solution Approach 1:
The patent segments the cache hierarchy into multiple smaller, distributed caches located close to each shader core rather than one large centralized cache. Each shader core has its own private L1 cache and potentially shared L2 cache in close proximity, eliminating the latency penalty of accessing a distant large cache while maintaining adequate capacity through the combined hierarchy across multiple cores.
Solution Approach 2:
The patent transitions from a single-dimension centralized cache architecture to a multi-dimensional distributed cache architecture. Instead of expanding cache size in one direction (which increases distance from cores), the system distributes cache capacity across multiple dimensions - multiple cores, multiple cache levels, and multiple cache instances - thereby increasing total capacity without increasing access latency.
Data Source
AI summary
Embodiments described herein are generally directed to improvements relating to power, latency, bandwidth and/or performance issues relating to GPU processing/caching. According to one embodiment, a state of multiple intellectual property (IP) cores that have access to a common cache via a central fabric is observed. Responsive to the observed state being indicative of performance of a standalone workload by a first IP core of the multiple IP cores, the common cache is treated as a local cache of the first IP core by powering off the central fabric and causing the first IP core to access the common cache via a low power access path between the first IP core and the common cache that is outside of the central fabric.