Neural Network Compute Tile With Separate Activation And Parameter Banks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently performing tensor computations for deep neural networks due to high instruction bandwidth and memory requirements, limiting computational efficiency and scalability.
Innovation Solution
A computing unit with a first memory bank of narrow data width for input activations and a second memory bank of wider data width for parameters, coupled with MAC operators, allows for efficient tensor computations by distributing computations across multiple tiles and exploiting memory hierarchy for high utilization of MAC operators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single memory bank is used to store both input activations and parameters, then memory access is simplified, but the data width must be large enough to accommodate both types of data, increasing memory bandwidth requirements and reducing computational efficiency
Solution Approach 1:
The memory system is segmented into two separate memory banks: a first memory bank for storing input activations and a second memory bank for storing parameters. This segmentation allows each memory bank to be optimized for its specific data type, with the first memory bank having a data width matched to activation requirements and the second memory bank having a data width matched to parameter requirements, thereby reducing overall memory bandwidth requirements while maintaining high computational efficiency
2Quantity of substance
If a narrow data width memory is used for input activations, then memory bandwidth is reduced, but the data width is insufficient to efficiently store and access parameters
Solution Approach 1:
The memory system is segmented into two separate memory banks: a first memory bank for storing input activations and a second memory bank for storing parameters. This segmentation allows each memory bank to be optimized for its specific data type, with the first memory bank having a narrower data width suitable for activations and the second memory bank having a wider data width suitable for parameters, thereby reducing overall memory bandwidth requirements while maintaining high computational efficiency
Solution Approach 2:
Each memory bank is designed with local quality optimization: the first memory bank is optimized with a data width specifically matched to input activation requirements, while the second memory bank is optimized with a data width specifically matched to parameter requirements. This local optimization ensures that each memory bank operates at peak efficiency for its specific data type without being constrained by the requirements of the other data type
Data Source
AI summary
A computing unit is disclosed, comprising a first memory bank for storing input activations and a second memory bank for storing parameters used in performing computations. The computing unit includes at least one cell comprising at least one multiply accumulate (“MAC”) operator that receives parameters from the second memory bank and performs computations. The computing unit further includes a first traversal unit that provides a control signal to the first memory bank to cause an input activation to be provided to a data bus accessible by the MAC operator. The computing unit performs one or more computations associated with at least one element of a data array, the one or more computations being performed by the MAC operator and comprising, in part, a multiply operation of the input activation received from the data bus and a parameter received from the second memory bank.