Neural Network Bottleneck-Layer Circuit for Reduced Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks, particularly deep CNNs, face significant computational and bandwidth challenges due to large data transfers between layers, leading to high power consumption and inefficient memory usage in existing hardware accelerators like GPUs and NPUs.
Innovation Solution
A hardware accelerator circuit with a custom data path that includes on-chip memory to store intermediate results, reducing off-chip memory accesses by processing up to three bottleneck layers in parallel, balancing memory usage with computational overhead, and co-locating arithmetic and logical operators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large data transfers between layers are performed in bottleneck stages, then computational accuracy is maintained, but power consumption increases and memory efficiency decreases
Solution Approach 1:
The bottleneck stage is segmented into multiple sub-stages (first bottleneck sub-stage, second bottleneck sub-stage, third bottleneck sub-stage) that process data in sequential steps. This segmentation allows intermediate results to be retained within on-chip memory structures rather than requiring complete data transfers, thereby maintaining computational accuracy while reducing power consumption associated with off-chip memory accesses.
Solution Approach 2:
On-chip memory structures (first intermediate result memory, second intermediate result memory, third intermediate result memory) are introduced as intermediary storage between processing stages. These intermediaries hold partial computation results, eliminating the need for complete data transfers between bottleneck stages and reducing power consumption while preserving computational integrity.
2Use of energy by moving object
If on-chip memory is used for intermediate results, then power consumption is reduced, but device complexity increases
Solution Approach 1:
The on-chip memory structures serve multiple functions: they store intermediate results from previous stages, provide input data for subsequent stages, and enable parallel processing operations. This multi-functionality reduces the need for separate dedicated memory components, thereby limiting the increase in device complexity while achieving power reduction benefits.
Solution Approach 2:
Multiple memory structures are nested within the bottleneck stage hierarchy, with each sub-stage having its dedicated intermediate result memory. This nested organization allows efficient local access to intermediate data while maintaining a structured, manageable complexity architecture rather than requiring a flat, monolithic memory system.
3Productivity
If parallel processing is enabled across bottleneck layers, then processing speed increases, but memory bandwidth requirements increase
Solution Approach 1:
The parallel processing capability is achieved by segmenting the bottleneck stage into multiple independent sub-stages that can operate simultaneously on different data streams. Each sub-stage processes a portion of the input data independently, enabling parallel execution while each sub-stage uses its own dedicated intermediate memory, thereby avoiding the need for increased total memory bandwidth.
Solution Approach 2:
Parallel processing is achieved by adding a temporal dimension to the data flow, where multiple sub-stages process different time steps or data batches concurrently. This dimensional transformation allows parallelism without requiring proportional increases in memory bandwidth, as each sub-stage operates with its own memory resources in a time-multiplexed manner.
Data Source
AI summary
Methods and apparatuses herein improve bottleneck-layer processing in neural networks. Example advantages include reducing the number of accesses needed to external memory, allowing processing to run in parallel in successive bottleneck layers, based on the use of partial convolutional results, and balancing the amount of “local” memory used for storing convolutional results against the computational overhead of recomputing partial results. One aspect of the methods and apparatuses involves co-locating arithmetic and logical operators and temporary storage in the same data path, with the approach yielding both higher performance and greater energy efficiency in the implementation of bottleneck layers for neural network processing.