Dual Arithmetic Modules for Inference Device Memory Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing inference devices face limitations in processing speed due to memory bandwidth bottlenecks when performing convolutional neural network operations on multi-channel image data, particularly in battery-driven devices like digital cameras.
Innovation Solution
The proposed inference device employs a dual arithmetic module architecture, where the first module processes image data in larger chunks with fewer channels and filters, and the second module processes smaller chunks with more channels and filters in parallel, utilizing line memories optimized for the filter sizes and channel counts, allowing for pipeline processing and reducing memory bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single arithmetic module processes multi-channel image data with convolution operations, then the device can perform inference, but memory bandwidth bottlenecks occur reducing processing speed
Solution Approach 1:
The single arithmetic module is divided into multiple arithmetic modules (first arithmetic module and second arithmetic module). Each module processes different portions of the multi-channel image data in parallel, reducing the memory bandwidth requirement for each individual module while maintaining overall processing throughput.
Solution Approach 2:
The patent introduces a new dimension of parallel processing by adding multiple arithmetic modules that operate simultaneously on different channels or portions of the image data. This transforms the single-threaded memory access pattern into a multi-threaded parallel access pattern, effectively distributing the memory bandwidth burden.
2Productivity
If the number of arithmetic units is increased to process more channels in parallel, then processing speed improves, but device complexity increases
Solution Approach 1:
The arithmetic processing functionality is segmented into multiple independent arithmetic modules, each with its own arithmetic units. This allows the system to process multiple channels in parallel while keeping each individual module relatively simple and manageable.
Solution Approach 2:
Each arithmetic module is designed with universal functionality to handle convolution operations on multi-channel data. The modules can process different numbers of channels and filters based on the input data requirements, providing flexibility without requiring dedicated hardware for each specific configuration.
3Productivity
If line memories are optimized for specific filter sizes and channel counts, then processing efficiency increases, but adaptability to different data configurations decreases
Solution Approach 1:
The line memories are designed with dynamic configuration capabilities, allowing their parameters (such as depth and width) to be adjusted based on the input data characteristics. The system can adaptively allocate memory resources to match the specific filter sizes and channel counts of the incoming image data, maintaining both efficiency and adaptability.
Solution Approach 2:
The patent employs parameter changes in the line memory configuration to optimize processing for different data formats. By dynamically adjusting memory parameters such as depth, width, and organization, the system can efficiently handle various filter sizes and channel counts without requiring completely different memory architectures for each case.
Data Source
AI summary
An inference device includes a first arithmetic module and a second arithmetic module that execute arithmetic processing including a convolution process and a pooling process. The first arithmetic module includes a first memory that stores a plurality of first row data items generated by dividing first image data for each first number of pixels in a row direction and a plurality of first arithmetic units that execute a first convolution process on the plurality of first row data items. The second arithmetic module includes a second memory that stores a plurality of second row data items generated by dividing second image data for each second number of pixels in the row direction and a plurality of second arithmetic units that execute a second convolution process on the plurality of second row data items. The first image data and the second image data have different numbers of channels.