GPU Streaming Buffer Bypass for Lower Cache Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GPU processing scenarios face issues with power, latency, bandwidth, and performance, particularly in atomic reduction operations, shared caching architectures, and last-level cache management, which impact scalability and efficiency.
Innovation Solution
Implementing a streaming buffer between producer and consumer IP cores to bypass the mid-level cache, using a double buffering technique, and optimizing cache structures to reduce power consumption and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If atomic reduction is performed in shared local memory or L3 cache, then the operation can be completed, but bandwidth is consumed and latency is created
Solution Approach 1:
The patent segments the atomic reduction operation into two parts: a first portion executed in shared local memory and a second portion executed in the L3 cache. This segmentation allows the operation to leverage the speed of local memory for critical path operations while using the larger L3 cache for final aggregation, thereby reducing overall latency and bandwidth consumption compared to performing the entire operation in a single location.
2Productivity
If video tiles are written to DRAM for AI inferences, then the rendering can be done separately, but huge bandwidth and latency costs are incurred
Solution Approach 1:
The patent applies preliminary action by maintaining tile data in the L3 cache after rendering, rather than immediately writing to DRAM. The cache retention policy keeps frequently accessed tile data in the L3 cache, so that when AI inference operations need to access this data, it is already available in the fast cache memory, avoiding the high bandwidth cost of repeated DRAM accesses.
3Adaptability or versatility
If a mid-level cache is shared by multiple IP cores, then cache resources are utilized, but cache power and central fabric power are consumed for standalone workloads
Solution Approach 1:
The patent implements a dynamic cache access mechanism where the L3 cache can be selectively accessed by different IP cores based on workload requirements. For standalone workloads, the cache access is dynamically routed to avoid unnecessary power consumption in the central fabric and other IP cores. The cache controller dynamically manages which cores can access the cache and at what power levels, allowing the system to adapt power consumption to actual usage patterns.
4Quantity of substance
If the L3 cache is made larger to improve caching, then cache capacity increases, but it gets further away from shader cores driving up power and latency
Solution Approach 1:
The patent segments the cache hierarchy into multiple levels with different capacities and performance characteristics. Rather than creating one large L3 cache far from the shader cores, the system uses smaller, distributed cache structures that are closer to the compute units. This segmentation allows the system to provide adequate cache capacity through aggregation of multiple smaller caches while maintaining low latency and power consumption by keeping each cache segment physically close to the shader cores that need it.
Data Source
AI summary
Embodiments described herein are generally directed to improvements relating to power, latency, bandwidth and/or performance issues relating to GPU processing/caching. According to one embodiment, a system includes a producer intellectual property (IP) (e.g., a media IP), a compute core (e.g., a GPU or an AI-specific core of the GPU), a streaming buffer logically interposed between the producer IP and the compute core. The producer IP is operable to consume data from memory and output results to the streaming buffer. The compute core is operable to perform AI inference processing based on data consumed from the streaming buffer and output AI inference processing results to the memory.