On-Chip Memory Fusion for Neural Network Accelerator DRAM Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators for neural networks face inefficiencies in processing due to excessive DRAM reads and writes for intermediate values, leading to performance slowdowns, as they often require accessing device memory for operations that could be optimized by storing tensors in on-chip memory.
Innovation Solution
A fusion strategy is implemented that merges multiple operations into one large operation, storing tensors associated with operations in on-chip memory instead of device memory, optimizing DRAM accesses and allowing for greater operational intensity by strategically utilizing on-chip memory like SRAM-based Global Memory, and extending traditional operation fusion to combine multiple layers given sufficient on-chip capacity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tensors are stored in off-chip device memory, then memory capacity is sufficient, but excessive DRAM reads and writes cause performance slowdowns
Solution Approach 1:
The patent segments memory storage into two distinct locations: on-chip memory (SRAM) and off-chip memory (DRAM). By dividing the memory hierarchy, the system can store frequently accessed tensors in fast on-chip memory while keeping less frequently accessed data in off-chip memory, thereby reducing the time spent on DRAM accesses and improving overall processing speed.
Solution Approach 2:
The system performs preliminary action by pre-loading tensors into on-chip memory before they are needed for computation. This advance preparation ensures that when computations occur, the required data is already in fast on-chip memory, eliminating the need for time-consuming DRAM reads during the actual processing operations.
2Productivity
If multiple operations are merged into one large operation, then operational intensity increases, but on-chip memory capacity requirements increase
Solution Approach 1:
The patent applies merging by combining multiple separate operations into one large fused operation. This increases operational intensity by keeping more data in on-chip memory and performing multiple computations in a single operation, reducing the need for repeated data transfers and improving overall efficiency despite the increased on-chip memory requirements.
3Productivity
If fusion strategy is applied, then DRAM accesses are reduced, but device complexity increases
Solution Approach 1:
The system implements dynamic fusion strategies that can adapt to different computational workloads and data patterns. The fusion approach is not fixed but can be adjusted based on the specific requirements of each operation, allowing the system to optimize DRAM access reduction while managing device complexity through flexible, context-aware memory management.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining a hardware datapath for a hardware accelerator computer chip.