Runtime Prefetcher Modulation for High Memory Bandwidth

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Modern hardware prefetchers in microprocessors degrade performance and increase memory latency at high memory bandwidth utilization levels, particularly in large data centers, necessitating a dynamic solution to balance cache miss rates and memory latency.

Innovation Solution

A system that dynamically modulates hardware prefetchers based on memory bandwidth utilization thresholds, enabling or disabling them to maintain optimal performance by using system software to adjust prefetcher states based on observed memory bandwidth levels.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If hardware prefetchers are enabled to reduce cache miss rates, then cache hit rate is improved, but memory latency increases at high memory bandwidth utilization levels

Engineering Contradiction:

Improvecache hit rateVSAvoidmemory latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The prefetcher is transformed from a static always-on component to a dynamic component whose behavior changes based on memory bandwidth utilization conditions. The system continuously monitors memory bandwidth usage and adjusts prefetcher enablement state accordingly, making the prefetcher adaptive to changing workload conditions rather than operating in a fixed state

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes the operational parameters of the prefetcher based on memory bandwidth utilization thresholds. When bandwidth utilization exceeds the upper threshold, the prefetcher is disabled; when it falls below the lower threshold, the prefetcher is re-enabled. This parameter-based control allows optimization of both cache hit rate and memory latency under different bandwidth conditions

Inventive Principle:

Principle #35Parameter changes

2Loss of time

If hardware prefetchers are disabled to reduce memory latency at high bandwidth utilization, then memory latency is reduced, but cache miss rates increase

Engineering Contradiction:

Improvememory latencyVSAvoidcache miss rate

Core Design Contradiction:

Loss of timeVSReliability

Solution Approach 1:

The system dynamically adjusts prefetcher behavior based on real-time memory bandwidth conditions. Rather than permanently disabling the prefetcher, the system enables it during low-bandwidth periods and disables it only during high-bandwidth periods when it would harm performance, creating a dynamic adaptation strategy

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The operational state of the prefetcher is changed based on bandwidth threshold parameters. The system uses upper and lower bandwidth thresholds to control when the prefetcher should be disabled or enabled, allowing optimization of memory latency while accepting increased cache misses only when bandwidth conditions warrant such trade-offs

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If hardware prefetchers are frequently toggled on and off to adapt to changing bandwidth conditions, then adaptability is improved, but system stability deteriorates due to constant state changes

Engineering Contradiction:

Improveprefetcher adaptabilityVSAvoidsystem stability

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The system establishes upper and lower bandwidth thresholds in advance to determine when prefetcher state changes should occur. By pre-defining these threshold boundaries, the system prevents arbitrary or frequent toggling and ensures that state changes only happen when bandwidth conditions clearly warrant them, thereby maintaining stability while preserving adaptability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system continuously monitors memory bandwidth utilization and uses this feedback to control prefetcher enablement. The feedback mechanism compares current bandwidth usage against predefined thresholds and only triggers state changes when the feedback indicates sustained high or low bandwidth conditions, preventing unnecessary toggling and promoting system stability

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12493556B2Hardware control system to modulate prefetchers based on runtime telemetry

Publication Date: 2025.12.09 GOOGLE LLC

AI summary

Generally disclosed herein is an approach for dynamically modulating a hardware prefetcher using system software. The hardware prefetcher may be disabled when the amount of the memory bandwidth utilization level is above an upper threshold. The hardware prefetcher may be enabled when the amount of the memory bandwidth utilization level is below a lower threshold. To avoid constantly toggling the hardware prefetcher on and off and allow for smoother transitions, the amount of the memory bandwidth utilization level is observed until it stays above the upper threshold or below the lower threshold for a particular time before the system software changes the latest state of the hardware prefetcher.