AI Accelerator Instructions With Embedded Parameters for Faster Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI hardware accelerators face inefficiencies due to excessive memory access for parameter values, leading to increased operation overhead and reduced performance in neural network inference tasks.
Innovation Solution
The solution involves embedding fixed parameter values directly into instructions within the accelerator, converting loop variable indices to invariable indices through loop unrolling, and optimizing instruction caching by ensuring same parameter values are used across thread blocks, thereby reducing memory access and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If parameter values are loaded from external memory for each inference task, then the accelerator can perform flexible inference operations, but the memory access overhead increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-loading parameter values into the instruction cache before inference tasks are executed. The system identifies and loads parameter values that are likely to be used in upcoming inference operations, so that when inference tasks arrive, the parameters are already available in fast memory, eliminating the need for repeated external memory access during actual inference execution.
Solution Approach 2:
The patent merges parameter value storage with instruction storage by storing parameter values directly within the instruction cache structure. Instead of maintaining separate storage for parameters and instructions, the system combines them into a unified cache that can hold both instruction codes and their associated parameter values, reducing memory access overhead while maintaining flexibility.
2Quantity of substance
If parameter values are stored in external memory, then memory capacity is sufficient for large models, but access speed decreases due to external memory limitations
Solution Approach 1:
The patent segments parameter storage into two parts: frequently accessed parameter values are stored in the fast instruction cache, while less frequently accessed parameters remain in external memory. This segmentation allows the system to optimize access speed for critical parameters while maintaining sufficient total storage capacity in external memory for large models.
Solution Approach 2:
The patent implements a nested memory structure where the instruction cache (fast memory) is nested within the broader external memory system. The instruction cache acts as a small, fast subset of the larger external memory, creating a hierarchical storage system that provides both high-speed access for immediate needs and large-capacity storage for the complete model parameters.
3Speed
If the instruction cache is optimized for high speed, then inference execution is faster, but the cache size is reduced limiting parameter storage
Solution Approach 1:
The patent changes the parameters of the instruction cache by adjusting the granularity and organization of stored data. Instead of storing only traditional instruction codes, the cache is reconfigured to store compressed or efficiently encoded parameter values alongside instructions, allowing more parameter data to fit within the same physical cache capacity while maintaining high-speed access characteristics.
Data Source
AI summary
A processor-implemented accelerator method includes: reading, from a memory, an instruction to be executed in an accelerator; reading, from the memory, input data based on the instruction; and performing, on the input data and a parameter value included in the instruction, an inference task corresponding to the instruction.