GPU Load Store Cache Replacement for Prefetched Data Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cache technologies in graphics processing units (GPUs) are inefficient in handling prefetched data, leading to increased stalls and power usage due to premature invalidation of data before usage, especially in applications like machine learning that require significant data prefetching.
Innovation Solution
Implementing a prefetch aware LRU (Least Recently Used) replacement policy in the load store cache that differentiates between used and unused prefetched data, ensuring data is retained until utilized, thereby reducing the need for re-fetching from lower-level caches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a traditional LRU replacement policy is used in the load store cache, then the cache management is simple, but prefetched data is prematurely invalidated leading to increased stalls and power usage
Solution Approach 1:
The patent segments the cache replacement decision-making process by introducing a prefetch detection mechanism that identifies whether evicted data was prefetched or legitimately requested. This segmentation allows differentiated handling: prefetched data that hasn't been used is retained, while genuinely unused data is evicted, resolving the contradiction between simple cache management and reduced power consumption.
Solution Approach 2:
The patent performs preliminary action by detecting and marking prefetched data before replacement decisions are made. The prefetch detection mechanism proactively identifies prefetched entries and prevents their premature invalidation, allowing the cache to retain potentially useful data and avoid re-fetching operations that consume additional power.
2Productivity
If a traditional LRU replacement policy is used in the load store cache, then the cache structure is simple, but data is invalidated before usage requiring re-fetching from lower-level caches
Solution Approach 1:
The patent segments cache entries into prefetched and non-prefetched categories, applying different replacement policies to each. This segmentation prevents the invalidation of useful prefetched data, reducing cache misses and the need to re-fetch from lower-level caches, thereby reducing latency without complicating the overall cache structure.
Solution Approach 2:
The patent implements feedback by monitoring cache access patterns and using this information to inform replacement decisions. The prefetch detection mechanism provides feedback about which entries are prefetched, allowing the cache controller to make more informed decisions that prevent premature invalidation of useful data, thus reducing re-fetching latency.
3Loss of energy
If prefetched data is retained in the cache without differentiation, then power usage increases due to retaining unused data, but invalidating it causes re-fetching operations
Solution Approach 1:
The patent segments cache management into two distinct paths: one for prefetched data and one for legitimately requested data. By segmenting the replacement decision based on prefetch status, the system can selectively retain or evict data, avoiding the waste of power on storing unused prefetched data while preventing the inefficiency of re-fetching operations.
Solution Approach 2:
The patent changes the parameter used for replacement decisions from单纯的 recency (traditional LRU) to a composite criterion that includes prefetch status. This parameter change allows the cache to differentiate between prefetched and non-prefetched data, optimizing both power consumption and processing efficiency by making replacement decisions based on the actual likelihood of data usage.
Data Source
AI summary
Prefetch aware LRU cache replacement policy is described. An example of an apparatus includes one or more processors including a graphic processor, the graphics processor including a load store cache having multiple cache lines (CLs), each including bits for a cache line level (CL level) and one or more sectors for data storage; wherein the graphics processor is to receive one or more data elements for storage in the cache; set a CL level to track each CL receiving data, including setting CL level 1 for a CL receiving data in response to a miss in the cache and setting a CL level 2 for a CL receiving prefetched data in response to a prefetch request, and, upon determining that space is required in the cache to store data, apply a cache replacement policy, the policy being based at least in part on set CL levels for the CLs.