Convolution Engine Data Reuse for Lower DNN Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware implementations of deep neural networks (DNNs) face challenges in terms of high memory bandwidth requirements and power consumption due to inefficient data transfer for convolution operations, particularly in resource-constrained environments.
Innovation Solution
The proposed hardware implementation optimizes data transfer by providing input data and weights to convolution engines in an order that allows reuse across multiple cycles, reducing the need for repeated memory reads and thus lowering memory bandwidth and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If input data and weights are read from memory for each convolution operation, then processing capability is improved, but memory bandwidth requirement increases
Solution Approach 1:
The patent applies preliminary action by pre-loading input data and weights into on-chip buffers before convolution operations begin. The memory interface reads data in advance and stores it in local buffers, so that when convolution engines need the data, it is already available in fast on-chip memory rather than requiring repeated access to external memory for each operation.
Solution Approach 2:
The patent introduces on-chip buffers as intermediary storage between external memory and convolution engines. These buffers act as mediators that hold data temporarily, allowing multiple convolution operations to access the same data without requiring repeated memory reads. The buffers decouple the memory interface from the computation units, enabling efficient data reuse.
2Productivity
If input data and weights are repeatedly read from memory, then processing continuity is maintained, but power consumption increases
Solution Approach 1:
The patent ensures continuous processing by maintaining data in on-chip buffers throughout the convolution computation phase. Once data is loaded into buffers, multiple convolution engines can continuously access it without interruption or repeated memory access. This continuous access pattern eliminates idle time while avoiding the high power cost of repeated memory transactions.
Solution Approach 2:
The on-chip buffers serve as intermediaries that enable continuous processing by holding data locally. The buffers reduce power consumption by eliminating repeated activation of high-power memory interfaces, while still providing continuous data supply to convolution engines through fast on-chip access.
3Quantity of substance
If data is stored in on-chip buffers for reuse, then memory bandwidth is reduced, but silicon area increases
Solution Approach 1:
The patent applies partial action by implementing on-chip buffers with capacity optimized for specific convolution layer configurations rather than providing full dataset storage. The buffer size is tuned to hold exactly the amount of data needed for a given number of convolution operations, avoiding excessive area while still achieving significant memory bandwidth reduction for the target application.
Solution Approach 2:
The patent enables parameter changes by making buffer sizes and configurations adjustable based on the specific convolution layer being processed. Different convolution layers may require different buffer capacities, and the system can reconfigure buffer parameters dynamically or through compilation to match the requirements of each layer, optimizing the area-bandwidth tradeoff for each case.
4Productivity
If convolution engines process the same weights multiple times, then computation efficiency is improved, but data transfer overhead increases
Solution Approach 1:
The patent applies preliminary action by loading weights into on-chip buffers before the computation phase begins. Since the same weights are used across multiple convolution operations, they are pre-fetched from external memory and stored locally once, eliminating repeated data transfer overhead while enabling efficient reuse by multiple convolution engines.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Hardware implementations of, and methods for processing, a convolution layer of a DNN that comprise a plurality of convolution engines wherein the input data and weights are provided to the convolution engines in an order that allows input data and weights read from memory to be used in at least two filter-window calculations performed either by the same convolution engine in successive cycles or by different convolution engines in the same cycle. For example, in some hardware implementations of a convolution layer the convolution engines are configured to process the same weights but different input data each cycle, but the input data for each convolution engine remains the same for at least two cycles so that the convolution engines use the same input data in at least two consecutive cycles.