In-Flight Convolution Processing to Cut Memory Access Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hardware and methods are not well-suited for the increasing demands of speed, performance, and energy conservation required for computationally intensive machine learning applications, particularly due to excessive memory access and data movement during arithmetic operations like convolutions.
Innovation Solution
In-flight processing of pooling and convolution operations is performed without intermediate storage, combining these steps in parallel to reduce memory access and data movement, thereby reducing power consumption and computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional hardware and software methods are used for convolution operations, then compatibility with existing processors is maintained, but computational speed and power efficiency deteriorate due to excessive memory access and data movement
Solution Approach 1:
The patent merges pooling and convolution operations into a single integrated hardware unit that processes both operations simultaneously without intermediate storage. This combines multiple computational steps into one unified operation, eliminating the need to store and retrieve intermediate results between pooling and convolution, thereby significantly reducing memory access overhead and power consumption while maintaining high computational speed.
Solution Approach 2:
The patent performs pooling operations in advance as part of the convolution process itself, rather than as a separate subsequent step. By incorporating pooling within the convolution operation flow, the system eliminates the need for separate intermediate storage and retrieval operations, reducing both computational time and energy consumption.
2Productivity
If conventional scalar processors are used for matrix operations, then programming simplicity is maintained, but computational throughput deteriorates due to sequential processing steps
Solution Approach 1:
The patent segments the convolution operation into multiple parallel processing units, each handling different spatial locations or feature channels simultaneously. This segmentation enables parallel computation across multiple data elements, dramatically increasing throughput while keeping each individual processing unit relatively simple in structure.
Solution Approach 2:
The patent transitions from sequential scalar processing to parallel vectorized processing by introducing additional computational dimensions. Instead of processing one element at a time through sequential steps, the system processes multiple elements simultaneously across different dimensions (spatial locations, channels), thereby increasing throughput without proportionally increasing complexity.
3Loss of time
If software-based convolution operations are implemented on general-purpose processors, then programming flexibility is maintained, but execution time increases due to multiple memory access steps
Solution Approach 1:
The patent introduces a specialized hardware intermediary unit that sits between the memory and the processing units. This intermediary unit is specifically designed to handle the pooling and convolution operations with optimized data flow paths, eliminating the need for complex software algorithms to manage memory access patterns. The hardware intermediary automatically manages data retrieval and processing, reducing execution time while simplifying the software programming model.
Data Source
AI summary
In-flight operations in an inbound data path from a source memory to a convolution hardware circuit increase computational throughput when performing convolution calculations, such as pooling and element-wise operations. Various operations may be performed in-line within an outbound data path to a target memory. Advantageously, this drastically reduces extraneous memory access and associated read-write operations, thereby, significantly reducing overall power consumption in a computing system.