Hardware Accelerator Search Optimizing On-Chip Memory and Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators for neural networks face inefficiencies in processing due to excessive DRAM reads and writes for intermediate values, leading to performance slowdowns, as they often rely on off-chip memory for operations that could be optimized by storing tensors in on-chip memory.
Innovation Solution
A full-stack accelerator search technique that jointly optimizes the hardware datapath and fusion strategy, merging multiple operations into larger ones to minimize DRAM accesses by strategically utilizing on-chip memory, such as SRAM-based Global Memory, to store input, output, and weight tensors, thereby reducing the need for off-chip memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If tensors are stored in off-chip memory, then memory capacity is sufficient, but memory access latency increases and performance decreases
Solution Approach 1:
The memory system is segmented into multiple levels: fast on-chip memory (SRAM/DRAM) for frequently accessed tensors and slower off-chip memory for less frequently accessed data. This hierarchical segmentation allows the system to balance between access speed and storage capacity by placing different tensors in different memory levels based on their access patterns.
Solution Approach 2:
The patent implements a nested memory hierarchy where on-chip memory is nested within the broader off-chip memory system. The on-chip memory buffers and caches are embedded within the accelerator chip, creating a nested structure that provides fast access to critical tensors while maintaining sufficient total memory capacity through the outer off-chip memory layer.
2Productivity
If multiple operations are merged into one large operation, then DRAM accesses are minimized, but operational complexity increases
Solution Approach 1:
The patent merges multiple sequential operations into fused operations that can be executed in parallel within the tensor processing units. By combining operations such as matrix multiplications, convolutions, and activation functions into single fused kernels, the system increases operational intensity and reduces the number of separate memory access cycles required.
Solution Approach 2:
The system performs preliminary actions by pre-loading tensors into on-chip memory buffers before execution and pre-computing operation schedules. This preliminary preparation allows fused operations to execute efficiently without dynamic memory access decisions during computation, reducing operational complexity while maintaining high productivity.
3Loss of time
If on-chip memory is used to store tensors, then memory access latency is reduced, but chip area increases
Solution Approach 1:
The patent applies local quality by providing different memory configurations to different processing units based on their specific requirements. Not all tensor processing units are equipped with the same amount of on-chip memory; instead, memory resources are locally optimized for each unit based on its computational patterns and access requirements, balancing latency reduction with area constraints.
Solution Approach 2:
The system implements partial action by providing on-chip memory for only the most frequently accessed tensors rather than all tensors. The memory allocation is partial and selective, focusing on-cach eing critical activation tensors and weights that benefit most from fast access, while less critical data remains in off-chip memory, thus avoiding excessive chip area usage.
4Productivity
If fusion strategy is optimized for specific neural networks, then performance improves, but adaptability to different networks decreases
Solution Approach 1:
The patent implements a dynamic fusion strategy that adapts to different neural network architectures and workloads. The system can dynamically adjust which operations are fused and which tensors are cached in on-chip memory based on the specific characteristics of the loaded neural network, allowing optimization for each network while maintaining versatility across different network types.
Solution Approach 2:
The tensor processing units are designed with universal functionality that can execute various neural network operations through a common fused operation framework. The same hardware infrastructure supports multiple fusion strategies and can be reconfigured via software to optimize different network architectures, achieving both high performance and broad adaptability.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining a hardware datapath for a hardware accelerator computer chip.