Hardware-Guided Software Prefetching for Irregular Memory Accesses

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Hardware prefetchers struggle with irregular memory accesses and require a warm-up period to identify memory access patterns, while software prefetchers face challenges in identifying the correct time to prefetch instructions.

Innovation Solution

A system and method for hardware-guided software prefetch insertion that identifies target code locations based on hardware prefetcher performance, using ablation studies to determine prefetch-friendly and prefetch-unfriendly functions, and generates software prefetching instructions without a warm-up period.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If hardware prefetchers are used to automatically prefetch data, then data loading can occur ahead of use without programmer intervention, but the hardware prefetchers require warm-up periods to identify memory access patterns and cannot effectively handle irregular memory accesses

Engineering Contradiction:

Improvedata prefetching capabilityVSAvoidwarm-up period

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent performs preliminary analysis during compilation to identify memory access patterns and insert prefetch instructions directly into the code, eliminating the need for runtime warm-up periods. The compiler statically determines which data should be prefetched and when, based on analysis of the program's memory access behavior.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces software prefetch instructions as an intermediary mechanism between the compiler and hardware execution. These software-generated prefetch instructions bridge the gap by providing explicit prefetch directives to the hardware prefetch units, enabling effective prefetching without requiring the hardware to autonomously learn patterns during runtime.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If software prefetchers are used to insert prefetch instructions directly in programs, then no warm-up period is required and irregular memory accesses can be handled, but identifying the correct time to prefetch instructions is challenging

Engineering Contradiction:

Improvehandling of irregular memory accessesVSAvoididentification of correct prefetch timing

Core Design Contradiction:

Adaptability or versatilityVSDifficulty of detecting and measuring

Solution Approach 1:

The compiler performs preliminary static analysis of the program during compilation to identify memory access patterns and determine the optimal timing for prefetch operations. This advance planning allows the generated code to contain precisely positioned prefetch instructions that execute at the correct moments without requiring runtime decision-making.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent employs feedback mechanisms where the compiler analyzes program execution characteristics and memory access patterns, then uses this information to generate optimized prefetch instructions. The system continuously refines prefetch timing based on observed access patterns, ensuring prefetch operations occur at optimal moments.

Inventive Principle:

Principle #23Feedback

3Extent of automation

If hardware prefetchers examine memory access streams to identify patterns, then automatic prefetching can be achieved, but the prefetchers are incapable of handling irregular memory accesses

Engineering Contradiction:

Improveautomatic pattern identificationVSAvoidhandling of irregular memory accesses

Core Design Contradiction:

Extent of automationVSAdaptability or versatility

Solution Approach 1:

The patent segments memory access patterns into regular and irregular categories, applying different prefetching strategies to each. For regular patterns, hardware prefetchers continue to operate autonomously. For irregular patterns, the compiler inserts targeted software prefetch instructions that are specifically tailored to the irregular access behavior, thus handling both types effectively.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the operational parameters of prefetching by switching between hardware-driven automatic prefetching and compiler-directed software prefetching based on the characteristics of memory accesses. This parameter change allows the system to adapt to different access patterns, maintaining high effectiveness for both regular and irregular accesses.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12423101B1Hardware guided software prefetch insertion

Publication Date: 2025.09.23 GOOGLE LLC

AI summary

Generally disclosed herein is an approach for identifying target code locations for software prefetching instructions based on hardware prefetching information. The target code locations can be identified based on the locations where the hardware prefetchers were designed to prefetch loads. Hardware cache events related to processor events for different caches can be monitored for cache misses in a system running when hardware prefetchers are active and inactive. Code locations that may benefit from the use of the hardware prefetchers may be determined based on an ablation study. The identified code locations may be treated as recognized memory access patterns. Software prefetchers can be configured to use these memory access patterns for prefetching data while avoiding an extended time required to identify memory access patterns (i.e. a warm-up period) and fetching in locations where hardware prefetching is beneficial.