Inline Image Preprocessor for Flexible CNN Sample Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing convolution operations in neural networks are limited by the inability to perform strided or dilated convolutions due to conflicts between shift registers and data access patterns, leading to inefficiencies in memory usage and computational complexity.
Innovation Solution
An image preprocessor architecture that includes multiplexers, buffers, and shift registers to generate multiple data samples per clock cycle, allowing for efficient strided or dilated access patterns and supporting dynamic convolutional filter sizes, thereby optimizing convolution operations for neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If shift registers are used to cycle through samples in sequence, then the convolution operation can be performed using a DSP array, but strided or dilated convolutions cannot be performed due to conflict with data access patterns
Solution Approach 1:
The patent implements dynamic data access patterns by allowing the system to switch between sequential access (for standard convolutions) and strided/dilated access patterns (for advanced convolutions). The architecture dynamically adjusts memory addressing and data flow based on the convolution type being performed, enabling versatility without requiring separate dedicated hardware for each convolution type.
Solution Approach 2:
The patent segments the data access mechanism into multiple independent components: separate address generation units for different access patterns, modular memory bank structures, and configurable data routing logic. This segmentation allows each component to be optimized for specific convolution types while maintaining overall system flexibility.
2Productivity
If a large number of digital signal processors (DSPs) are used to perform convolution operations, then computational capability is improved, but the requirement for on-chip buffers and data access patterns becomes more complex
Solution Approach 1:
The patent merges the buffer memory with the DSP array structure, creating an integrated architecture where data can be streamed directly from memory banks to DSP units without requiring separate large on-chip buffers. This merging reduces the total memory requirement while maintaining high computational throughput.
Solution Approach 2:
The patent implements preliminary data preparation by pre-fetching and organizing input data into optimized formats before it reaches the DSP array. Data is pre-processed and arranged in memory banks according to the specific convolution operation requirements, reducing the buffering needs during the actual computation phase.
3Ease of manufacture
If sequential sample cycling through shift registers is used, then simple convolution is achieved, but memory bandwidth efficiency decreases for strided or dilated patterns
Solution Approach 1:
The patent changes the memory access parameters dynamically based on the convolution type. For strided or dilated convolutions, the system adjusts memory read addresses, data stride values, and transfer timing to optimize bandwidth utilization. This allows efficient memory usage while maintaining implementation simplicity through a unified architecture.
Data Source
AI summary
An example preprocessor circuit for formatting image data into a plurality of streams of image samples includes: a plurality of memory banks configured to store the image data; multiplexer circuitry coupled to the memory banks; a first plurality of registers coupled to the multiplexer circuitry; a second plurality of registers coupled to the first plurality of registers, outputs of the second plurality of registers configured to provide the plurality of streams of image samples; and control circuitry configured to generate addresses for the plurality of memory banks, control the multiplexer circuitry to select among outputs of the plurality of memory banks, control the first plurality of registers to store outputs of the second plurality of multiplexers, and control the second plurality of registers to store outputs of the first plurality of registers.