Processor Cache Eviction by Memory Reuse to Reduce DRAM Traffic

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional cache management policies fail to reduce off-chip memory traffic for transient data, such as activation data in neural network inference applications, leading to unnecessary data transfers between cache and DRAM.

Innovation Solution

Implementing a software-managed cache management policy using eviction priority controls based on memory reuse, which prioritizes data retention in cache through evict-last, evict-normal, and evict-first tags, allowing programmers to explicitly manage cache eviction based on the reuse factor of memory addresses.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of energy

If conventional cache management policies are used, then cache operation is simple and automatic, but off-chip memory traffic increases and cache hit rate decreases

Engineering Contradiction:

Improveoff-chip memory trafficVSAvoidcache management complexity

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The patent changes the cache management parameter from automatic/LRU-based to software-controlled eviction priority based on memory reuse patterns. By introducing eviction priority tags (evict-first, evict-normal, evict-last) that can be programmed based on memory address reuse characteristics, the system optimizes cache content selection to reduce off-chip memory traffic while maintaining manageable complexity through software-based control.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system enables software to actively manage cache eviction decisions based on memory reuse patterns rather than relying solely on hardware-based LRU mechanisms. The eviction priority controls allow the software to make informed decisions about which data to retain in cache based on actual memory access patterns, effectively making the cache management self-optimizing for the specific application workload.

Inventive Principle:

Principle #25Self-service

2Reliability

If data is retained in cache longer, then cache hit rate increases, but cache capacity for other data decreases

Engineering Contradiction:

Improvecache hit rateVSAvoidcache flexibility

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent applies different eviction priorities to different memory addresses based on their reuse characteristics. Instead of uniform cache management, the system assigns evict-first, evict-normal, or evict-last tags to different data based on their memory reuse patterns, allowing the cache to retain data with higher reuse factors longer while still making room for new data that needs to be prioritized for eviction.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The eviction priority in the cache is made dynamic rather than static. The system can change eviction priorities based on runtime memory access patterns and reuse factors, allowing the cache management strategy to adapt to varying workload characteristics. This dynamic adjustment enables the cache to optimize hit rates while maintaining flexibility to accommodate different data types and access patterns.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12436897B2Cache management using eviction priority based on memory reuse

Publication Date: 2025.10.07 NVIDIA CORP

AI summary

Apparatuses, systems, and techniques to manage a cache located on a processor of a computing system using eviction priority based on based on memory reuse. Memory addresses associated with a workload of an application executing using the processor are identified. An amount of reuse of the memory addresses corresponding to the workload is determined. A cache management policy for the workload is determined based on the amount of reuse. The cache management policy is applied to the cache.