Graphics Processor Instruction Prefetch via Thread Dispatch

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In computing systems, the latency associated with instruction and data cold starts can significantly impact the execution metrics, particularly for short-running workloads like machine learning inference jobs.

Innovation Solution

The proposed solution involves prefetching instructions and/or parameters or constants for kernels to be executed in parallel with thread dispatch, using data provided within a thread dispatch command, to reduce cold start latency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If thread dispatch commands are processed sequentially without prefetching, then device complexity is reduced, but instruction fetch time and cold start latency increase

Engineering Contradiction:

Improvecold start latencyVSAvoidprefetch mechanism complexity

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The patent implements prefetching of kernel instructions and parameters by utilizing data from thread dispatch commands before the actual kernel execution begins. The system extracts kernel identifiers and parameter information from incoming thread dispatch commands, prefetches the corresponding kernel code and parameters into buffer memory, and prepares them for immediate execution. This preliminary action significantly reduces cold start latency by ensuring that kernel instructions are already loaded and ready when needed, rather than fetching them from slower memory during execution.

Inventive Principle:

Principle #10Preliminary action

2Speed

If prefetching is implemented for all kernels, then instruction fetch time is reduced, but memory bandwidth consumption increases

Engineering Contradiction:

Improveinstruction fetch speedVSAvoidmemory bandwidth consumption

Core Design Contradiction:

SpeedVSUse of energy by moving object

Solution Approach 1:

The patent employs selective prefetching rather than prefetching all kernels unconditionally. The system uses a buffer memory to store recently accessed kernel instructions and parameters, and only prefetches kernels that are likely to be executed soon based on the thread dispatch command patterns. This partial action approach reduces memory bandwidth consumption compared to aggressive prefetching of all possible kernels, while still achieving significant instruction fetch speed improvements for the most commonly accessed kernels.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system dynamically adjusts prefetching parameters such as buffer size, prefetch distance, and eviction policies based on workload characteristics. By monitoring kernel access patterns and execution frequencies, the system optimizes the balance between instruction fetch speed and memory bandwidth usage, adapting to different workload types including machine learning inference jobs with varying kernel access patterns.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20250077232A1Instruction prefetch based on thread dispatch commands

Publication Date: 2025.03.06 INTEL CORP

AI summary

A graphics processing device is provided that includes a set of compute units to execute a workload, a cache coupled with the set of compute units, and circuitry coupled with the cache and the set of compute units. The circuitry is configured to, in response to a cache miss for the read from a first cache, broadcast an event within the graphics processor device to identify data associated with the cache miss, receive the event at a second compute unit in the set of compute units, and prefetch the data identified by the event into a second cache that is local to the second compute unit before an attempt to read the instruction or data by the second thread.