HBM CNN Architecture With Virtual Banks And On-Chip Buffer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks face capacity limitations in memory, high latency, and bandwidth constraints, particularly in hybrid DDR and on-chip memory systems, which are costly and inefficient for data movement.
Innovation Solution
A neural network architecture utilizing High Bandwidth Memory (HBM) with dedicated virtual banks for feature map data and on-chip memory for weight and bias data, eliminating data movement between memory banks, and incorporating an on-chip buffer for efficient data transfer between convolutional and depthwise units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If HBM is used to increase storage capacity and bandwidth, then memory capacity and bandwidth are improved, but latency increases and flexibility decreases
Solution Approach 1:
The HBM is segmented into multiple virtual banks (first virtual bank, second virtual bank, third virtual bank, fourth virtual bank) that can be independently accessed. This segmentation allows parallel data retrieval operations, reducing the effective latency experienced by the system while maintaining the high capacity and bandwidth benefits of HBM.
2Productivity
If data is stored in HBM virtual banks, then bandwidth and capacity are increased, but data movement between memory banks becomes a bottleneck
Solution Approach 1:
An on-chip buffer is introduced as an intermediary component between the HBM virtual banks and the processing units. The buffer receives data from any virtual bank and makes it available to the depthwise convolution unit, eliminating the need for data to physically move between HBM banks. This intermediary approach maintains high bandwidth while avoiding data movement bottlenecks.
3Device complexity
If depthwise layers are used to decrease computation, then computational complexity is reduced, but hardware utilization efficiency decreases
Solution Approach 1:
The depthwise convolution unit is designed with multi-functionality, capable of performing both standard convolution operations and depthwise convolution operations. This universal design allows the same hardware resources to be efficiently utilized for different types of neural network layers, maintaining high hardware utilization while supporting the computation-reduced depthwise layer architecture.
4Loss of time
If on-chip memory is used to provide high bandwidth and low latency, then bandwidth and latency are improved, but storage capacity becomes insufficient
Solution Approach 1:
The memory hierarchy is designed with a nested structure where the on-chip buffer is nested within the FPGA device, and the HBM virtual banks are nested within the HBM package. This nested architecture allows the system to leverage the high speed of on-chip memory for frequently accessed data while using the high capacity of HBM for storing large neural network datasets, effectively combining the advantages of both memory types.
Data Source
AI summary
A system includes a high bandwidth memory (HBM) and a convolutional neural network (CNN) engine. The HBM includes a virtual bank portion and a system memory portion. The virtual bank portion is configured to store a feature map data and the system memory portion is configured to support data exchanges with a host. The CNN engine includes a convolutional unit configured to execute convolutional layer instructions, a depthwise convolutional unit configured to execute depthwise layer instructions, and a first on-chip buffer. The first on-chip buffer is configured to receive and store the feature map data from the virtual bank portion or receive and store data results from the convolutional unit. The first on-chip buffer is further configured to send the feature map data or the data results from the convolutional unit to the depthwise convolutional unit for processing.