Hybrid Convolution Operation Reducing Memory Accesses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional convolution operations in computing devices require a large number of memory accesses and introduce complexity, leading to reduced device performance, increased power consumption, and impaired user experience, especially when processing data at block edges or in applications like convolutional neural networks.
Innovation Solution
A hybrid convolution operation that 'slides' input data in one dimension and output data in another, reducing memory accesses by requiring only two blocks of input memory alignment per block of output data, using two accumulators to hold partial accumulation values and employing techniques like zero padding and ping-pong methods to minimize additional memory accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional convolution operations process pixels at block edges by accessing multiple input blocks, then complete convolution results are achieved, but memory access count increases significantly
Solution Approach 1:
The patent segments the convolution process into two distinct phases: intra-block convolution for most pixels and inter-block convolution only for edge pixels. This segmentation allows the majority of pixels to be processed with minimal memory accesses while only the necessary edge pixels perform additional memory accesses to access adjacent blocks, thereby resolving the contradiction between convolution completeness and memory access efficiency.
Solution Approach 2:
The patent applies different processing strategies to different regions of the image block. Interior pixels use a simplified convolution that accesses only the current block, while edge pixels use a more complex convolution that accesses adjacent blocks. This local differentiation ensures that memory access overhead is incurred only where necessary for complete convolution results.
2Measurement precision
If multiple blocks of input data are accessed to generate single block of output data, then accurate convolution is achieved, but processor resources and power are consumed
Solution Approach 1:
The patent segments pixels into interior pixels and edge pixels, applying different memory access strategies to each segment. Interior pixels require only single-block access while edge pixels access multiple blocks, minimizing the total number of memory accesses and thereby reducing power consumption while maintaining convolution accuracy.
Solution Approach 2:
The patent performs partial memory accesses by only loading the necessary portions of adjacent blocks for edge pixel processing, rather than loading entire blocks. This partial action approach reduces the energy cost of memory operations while ensuring sufficient data is available for accurate convolution at block edges.
3Measurement precision
If memory accesses are performed for block edge processing, then complete output data is generated, but complexity of computing addresses increases
Solution Approach 1:
The patent segments the processing logic into distinct paths for interior and edge pixels. The interior pixel path uses simple address computation within the current block, while the edge pixel path activates additional logic for computing addresses in adjacent blocks. This segmentation keeps the majority of processing simple while handling complexity only where necessary.
Solution Approach 2:
The patent performs preliminary checks to identify edge pixels before executing the convolution operation. By pre-identifying which pixels require multi-block access, the system can prepare appropriate address computation logic in advance, avoiding complex address calculations for pixels that don't require them and simplifying the overall control flow.
4Productivity
If intermediate state of input data is maintained for reuse, then memory transfers are reduced, but device complexity increases for neural network applications
Solution Approach 1:
The patent segments the memory access pattern into a first set of accesses for loading input blocks and a second set of accesses for storing output blocks. By organizing memory operations into these distinct segments and allowing overlapping execution with computation, the system reduces total memory transfer time and complexity while maintaining high data reuse efficiency.
Data Source
AI summary
A device includes one or more processors configured to retrieve a first block of data, the data corresponding to array of values arranged along at least a first dimension and a second dimension, to retrieve at least a portion of a second block of the data, and to perform a first hybrid convolution operation that applies a filter across the first block and at least the portion of the second block to generate output data. The output data includes a first accumulated block and at least a portion of a second accumulated block. The one or more processors are also configured to store the first accumulated block as first output data. The portion of the second block is adjacent to the first block along the first dimension and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.