Flash Transformer Accelerator for DRAM and Data Bus Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional computing systems face bottlenecks in processing large language models (LLMs) due to limited DRAM capacity and slow data transfer from flash memory, leading to performance issues such as sluggish response times and system crashes.
Innovation Solution
A hardware accelerator with a neural computing unit and local memory, capable of parallel computations, offloads LLM operations from the host CPU, utilizing a bytecode execution mechanism that minimizes host intervention and optimizes data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the LLM is stored in DRAM for fast access, then the processing speed is improved, but the cost and power consumption increase significantly
Solution Approach 1:
The system divides the LLM storage and processing into two segments: frequently accessed model parameters are kept in fast DRAM, while less frequently accessed parameters are stored in slower but higher-density storage. This segmentation allows the system to achieve fast processing for critical operations while reducing overall power consumption and cost by not keeping the entire model in high-power DRAM.
2Quantity of substance
If the LLM is stored in Flash memory to reduce cost, then the storage capacity is improved, but the data transfer speed deteriorates
Solution Approach 1:
The system extracts the most frequently accessed model parameters from Flash memory and loads them into DRAM or high-speed cache memory before processing. This extraction strategy allows the system to maintain large storage capacity in Flash while ensuring that critical data is available at high speeds when needed, effectively decoupling total storage capacity from active data transfer speed.
3Device complexity
If the host processor handles all LLM computations, then the system complexity is reduced, but the productivity deteriorates
Solution Approach 1:
The system merges the host processor with specialized accelerators (such as GPUs, TPUs, or FPGAs) that are optimized for neural network computations. This merging creates a hybrid processing architecture where the host processor handles high-level control and data preparation, while the accelerators perform computationally intensive matrix multiplications and activations, thereby increasing overall productivity without excessively increasing system complexity.
4Speed
If parallel MAC units are added to increase computation speed, then the processing speed is improved, but the device complexity increases
Solution Approach 1:
The system designs parallel MAC units with universal functionality that can perform multiple types of neural network operations (different activation functions, various matrix dimensions, different precision formats) through a single unified architecture. This universality allows the parallel units to handle diverse computational tasks without requiring separate specialized hardware for each operation type, thereby increasing computation speed while controlling the growth of device complexity.
Data Source
AI summary
The present invention discloses a hardware computing device comprising a non-transitory computer-readable storage medium configured to store one or more computer-executable instructions; a host processing unit configured to determine whether and how artificial neural network model-based computations should be assigned to a second processing unit; and a hardware accelerator configured to execute one or more artificial neural network computations assigned by the host processing unit, wherein the hardware accelerator sequentially executes instructions in an instruction stream with a single instruction fetch by detecting a successive address of a next instruction.