Dilated CNN Convolution Control for Lower Memory Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolution operations in convolutional neural networks (CNNs) require high input/output bandwidth, and data reuse between adjacent operations is difficult in dilated CNNs, leading to increased computational load and overfitting.
Innovation Solution
A neural network device and method that includes a memory and a neural network processor with a fetcher, buffer, operator, and writer, controlled by a controller to perform convolution operations with controlled offsets and intervals based on the dilation rate, enabling data reuse and reducing redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If dilated convolution operations are performed with increased receptive field size, then the model's ability to capture long-range dependencies is improved, but memory traffic and computational load increase significantly
Solution Approach 1:
The patent segments the dilated convolution operation into multiple steps, where each step processes a portion of the feature map with controlled offsets and intervals. This segmentation allows the system to handle large receptive fields by breaking down the computational task into manageable chunks, reducing the memory traffic and computational load compared to processing the entire feature map in a single operation.
Solution Approach 2:
The controller pre-calculates and determines the offsets and intervals for the fetcher and writer based on the dilation rate before executing the convolution operation. This preliminary action optimizes the data access pattern, ensuring that input data is fetched and output data is written with minimal redundancy, thereby reducing memory traffic and computational overhead.
2Loss of energy
If data is fetched and written without redundancy using controlled offsets and intervals, then memory traffic is reduced, but the complexity of data access control increases
Solution Approach 1:
The patent introduces a controller as an intermediary component that manages the fetcher and writer operations. The controller uses the dilation rate to calculate appropriate offsets and intervals, mediating between the convolution operation requirements and the memory access patterns. This intermediary approach reduces memory traffic by eliminating redundant data access while centralizing the control logic in a dedicated component rather than dispersing it throughout the system.
3Productivity
If the fetcher and writer operate with different offsets and intervals, then data reuse is optimized, but the synchronization complexity between components increases
Solution Approach 1:
The controller implements a feedback mechanism where it monitors the dilation rate and dynamically adjusts the offsets and intervals for both the fetcher and writer. Based on the convolution operation parameters, the controller provides feedback to optimize data reuse patterns. This feedback loop allows the system to achieve efficient data reuse with different offsets and intervals while maintaining synchronization through centralized control, rather than requiring complex peer-to-peer synchronization between components.
Data Source
AI summary
A neural network device includes: a memory configured to store a first feature map and a second feature map; and a neural network processor configured to operate a neural network, and comprising: a fetcher configured to fetch input data from the first feature map of the memory; a buffer configured to store the input data; an operator configured to generate output data by performing a convolution operation between the input data and a kernel; a writer configured to write the output data in the second feature map of the memory; and a controller configured to control the fetcher to fetch the input data and control the writer to write the output data, according to one or more intervals and one or more offsets determined based on a dilation rate of the kernel in multiple steps.