AI Model Instruction Prefetching to Reduce Memory Traffic Delays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid increase in the size and computational demands of artificial intelligence models has led to memory performance lagging behind computing performance, resulting in increased processing time and costs for large-scale models, particularly due to memory traffic delays and cache misses.
Innovation Solution
A computer system design that includes an on-chip memory (SRAM) to prefetch instruction blocks based on predicted execution flow, minimizing cache misses by fetching instructions from the main memory to the on-chip memory and then to the instruction cache, using a direct memory access controller and address adjuster to optimize address mapping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If instructions are fetched directly from main memory to instruction cache, then memory access speed is limited by main memory bandwidth, but system complexity is reduced
Solution Approach 1:
The memory system is segmented into multiple levels: main memory, on-chip memory (scratchpad), and instruction cache. This segmentation allows instructions to be fetched from on-chip memory which has higher bandwidth than main memory, improving instruction fetch speed while maintaining a manageable memory hierarchy structure.
Solution Approach 2:
An on-chip memory (scratchpad memory) is introduced as an intermediary between main memory and instruction cache. This intermediary buffer stores frequently accessed instruction blocks, reducing direct access to main memory and improving overall instruction fetch speed without requiring a complex multi-level cache system.
2Loss of time
If instruction blocks are prefetched to on-chip memory, then memory traffic delay is reduced, but on-chip memory bandwidth is consumed
Solution Approach 1:
Instruction blocks are prefetched to on-chip memory in advance based on predicted execution flow (sequential execution or branch targets). This preliminary action ensures that instructions are available in the scratchpad before they are needed, reducing memory traffic delay and allowing parallel execution without consuming bandwidth during critical execution paths.
Solution Approach 2:
The system uses branch prediction and execution flow analysis to dynamically determine which instruction blocks to prefetch. This feedback mechanism optimizes prefetching decisions based on actual program behavior, reducing unnecessary prefetching and minimizing on-chip memory bandwidth consumption while maximizing the reduction of memory traffic delay.
3Productivity
If address adjustment is performed for on-chip memory mapping, then address translation efficiency is improved, but processor complexity increases
Solution Approach 1:
The system changes the address parameter representation by maintaining separate address spaces for main memory and on-chip memory. An address adjustor translates virtual addresses to physical addresses specific to the scratchpad memory region, improving address translation efficiency for on-chip memory accesses while keeping the translation logic simple and localized.
Solution Approach 2:
An address adjustor component is introduced as an intermediary in the address translation path. This dedicated unit handles address translation specifically for on-chip memory accesses, improving efficiency for this common operation while isolating the complexity to a single component rather than distributing it throughout the entire processor.
4Speed
If instruction fetching and data fetching are performed in parallel, then overall execution speed is improved, but memory bus contention increases
Solution Approach 1:
The on-chip memory serves as an intermediary that decouples instruction fetching from data fetching operations. Instructions can be fetched from and executed using on-chip memory storage, while data is fetched from main memory through the data bus. This separation allows parallel execution of instruction fetch and data access operations without causing memory bus contention.
Solution Approach 2:
The memory access paths are segmented into separate channels: one for instructions (through on-chip memory) and one for data (through main memory). This segmentation enables parallel instruction fetching and data accessing without interference, improving execution speed while avoiding the harmful effect of memory bus contention that would occur with unified memory access.
Data Source
AI summary
A computer system includes a processor including an instruction cache and an on-chip memory coupled to the instruction cache, and a main memory connected to the processor through a bus and loading instructions and data of the artificial intelligence model from storage of the computer system, wherein the processor is configured to: fetch an instruction block including at least some of the instructions of the artificial intelligence model loaded into the main memory to the on-chip memory when executing the artificial intelligence model; fetch at least some of the instructions included in the fetched instruction block from the on-chip memory to the instruction cache and execute the fetched instructions; check, when a next instruction of a currently executed instruction does not exist in the instruction cache, whether the next instruction exists in the on-chip memory; and fetch the next instruction from the on-chip memory to the instruction cache based on a check result.