Neural Network Superlayer Partitioning for Memory Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing systems face inefficiencies due to memory constraints and excessive external communications, leading to suboptimal bandwidth consumption and energy usage.
Innovation Solution
The neural network layers are partitioned into a sequence of superlayers, allowing for efficient use of on-chip memory and minimizing external communications by implementing a global scheduling process that optimizes memory working sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If neural network layers are processed using conventional memory management, then processing can be performed, but bandwidth consumption increases and energy usage becomes suboptimal due to excessive external communications
Solution Approach 1:
The patent segments the neural network computation into distinct phases (parameter loading phase and activation processing phase) and organizes layers into superlayers. This segmentation allows for optimized memory access patterns where parameters are loaded once for multiple activations, reducing external communications and improving energy efficiency while managing bandwidth consumption systematically.
2Measurement precision
If parameters are loaded into memory for each layer individually, then processing accuracy is maintained, but external communications increase leading to higher energy usage
Solution Approach 1:
The patent applies preliminary action by loading all parameters for a superlayer into memory before processing any activations through those layers. This preliminary parameter loading ensures that subsequent activation processing can proceed with minimal external communications, maintaining processing accuracy while significantly reducing energy consumption from repeated memory accesses.
3Productivity
If conventional layer-by-layer processing is used, then implementation is straightforward, but memory utilization is suboptimal and processing efficiency decreases
Solution Approach 1:
The patent segments the neural network into superlayers (groups of consecutive layers) and processes them in a coordinated manner. This segmentation enables better memory utilization by loading parameters for entire superlayers at once and managing activation data flow across multiple layers simultaneously, improving processing efficiency while the segmentation itself provides a manageable structure for memory control.
Solution Approach 2:
The patent introduces a new dimension to traditional layer-by-layer processing by organizing layers into superlayers and implementing a global scheduling process that operates across this additional organizational level. This dimensional change from single-layer to superlayer processing enables more efficient memory utilization and improved throughput, with the added structure providing clear guidelines for memory management.
Data Source
AI summary
A computer-implemented method includes receiving a batch of neural network inputs to be processed using a neural network on a hardware circuit. The neural network has multiple layers arranged in a directed graph and each layer has a respective set of parameters. The method includes determining a partitioning of the neural network layers into a sequence of superlayers. Each superlayer is a partition of the directed graph that includes one or more layers. The method includes processing the batch of inputs using the hardware circuit, which includes, for each superlayer in the sequence: i) loading the respective set of parameters for the layers in the superlayer into memory of the hardware circuit, and ii) for each input in the batch, processing the input through each of the layers in the superlayer using the parameters in the memory of the hardware circuit to generate a superlayer output for the input.