Inline Image Preprocessor for Strided and Dilated Convolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing convolution operations in neural networks are limited by the inability to perform strided or dilated convolutions due to conflicts between shift registers and data access patterns, leading to inefficiencies in memory usage and computational complexity.
Innovation Solution
An image preprocessor architecture that includes multiplexers, buffers, and shift registers to generate multiple data samples per clock cycle, allowing for efficient strided or dilated access patterns and supporting dynamic convolutional filter sizes, thereby enabling efficient convolution operations with minimal buffers and memory bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If shift registers are used to cycle through samples in sequence, then the architecture is simple, but strided or dilated convolutions cannot be performed
Solution Approach 1:
The input image is divided into multiple tiles, and each tile is further divided into blocks that are processed in parallel through the DSP array. This segmentation enables the architecture to handle strided and dilated convolutions by organizing data access patterns into manageable segments that can be cycled through the shift registers efficiently.
Solution Approach 2:
The patent introduces a new dimension of parallelism by processing multiple tiles and blocks simultaneously through the DSP array. This multi-dimensional data organization (tiles × blocks × samples) allows the system to support complex convolution patterns while maintaining efficient memory access through the shift register architecture.
2Productivity
If a large number of DSPs are used to perform convolution operations, then computational capability is improved, but on-chip buffer requirements and data access complexity increase
Solution Approach 1:
The shift register architecture enables continuous cycling of data samples through the DSP array without interruption. Samples are fed continuously into the DSPs, allowing the computation to proceed without idle cycles and maximizing the utilization of the computational resources while minimizing buffer requirements.
Solution Approach 2:
The patent processes data in blocks and tiles, where each block contains more samples than strictly necessary for a single convolution operation. This partial over-provisioning of data blocks allows the shift registers to be fully utilized and enables the system to achieve high throughput with minimal buffer memory.
3Loss of energy
If shift registers are used for sequential sample access, then memory bandwidth is reduced, but the ability to perform strided or dilated convolutions is lost
Solution Approach 1:
The patent implements dynamic control of the shift register cycling through a state machine that adjusts the data access pattern based on the convolution type (standard, strided, or dilated). This dynamic reconfiguration allows the same hardware architecture to efficiently support multiple convolution patterns while maintaining low memory bandwidth consumption through the shift register mechanism.
Solution Approach 2:
The shift register architecture is designed to be universal, handling standard, strided, and dilated convolutions through a single unified data access mechanism. By controlling how samples are cycled through the registers, the system can adapt to different convolution patterns without requiring separate specialized hardware for each type.
Data Source
AI summary
An example preprocessor circuit for formatting image data into a plurality of streams of image samples includes: a plurality of memory banks configured to store the image data; multiplexer circuitry coupled to the memory banks; a first plurality of registers coupled to the multiplexer circuitry; a second plurality of registers coupled to the first plurality of registers, outputs of the second plurality of registers configured to provide the plurality of streams of image samples; bank address and control circuitry coupled to control inputs of the plurality of memory banks, the multiplexer circuitry, and the first plurality of registers; output control circuitry coupled to control inputs of the second plurality of registers; and a control state machine coupled to the bank address and control circuitry and the output control circuitry.