Neural Network Operator Cascades for Lower Peak Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face challenges in processing due to limited memory capacity in processors, which often need to share memory with other processes and hardware, necessitating efficient memory resource management without excessive usage.
Innovation Solution
A method is introduced to represent the neural network as a linear sequence of operators, identifying those that cannot form part of a cascade and processing them with the entire input and output feature maps in storage, while forming cascades of successive operators with partial input feature maps to reduce memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire input feature map and output feature map are stored in memory for processing each operator, then processing accuracy is maintained, but memory usage increases excessively
Solution Approach 1:
The patent segments the input feature map into multiple tiles or partitions, processing each tile independently through the operator sequence. This allows the system to load only the current tile into memory rather than the entire feature map, significantly reducing peak memory requirements while maintaining processing accuracy for each segment
Solution Approach 2:
The patent performs preliminary identification of operator cascades that can be executed without accessing the full feature map. By pre-analyzing the operator graph to determine which operators form self-contained cascades, the system can process these cascades using only local tile data, avoiding the need to load entire feature maps into memory
2Productivity
If memory capacity is increased to handle entire feature maps, then processing speed improves, but device complexity and resource allocation efficiency deteriorate
Solution Approach 1:
By dividing the feature map into tiles, the patent enables parallel processing of multiple tiles through different operator cascades. This segmentation allows efficient utilization of limited memory resources while maintaining high processing throughput, as multiple tiles can be processed in parallel or sequential order without requiring the entire feature map to reside in memory simultaneously
Solution Approach 2:
The patent processes operators in partial passes through the feature map tiles rather than requiring complete feature map loading. By executing operator cascades that operate on subsets of the feature map (partial action), the system achieves efficient processing with reduced memory capacity requirements, avoiding the need for excessive memory allocation
3Ease of operation
If operators are processed independently with full feature maps, then processing simplicity is maintained, but memory efficiency and resource utilization worsen
Solution Approach 1:
The patent merges adjacent operators in the operator graph that can be executed sequentially on the same feature map tile, forming operator cascades. This merging allows the system to process multiple operators in a single memory load cycle, improving memory efficiency without significantly complicating the processing logic, as the cascades are identified and executed as unified operation sequences
Data Source
AI summary
A method reduces storage usage during processing of a neural network performed by an information processing apparatus comprising a storage. The network may be represented by operators that operate on an input feature map and generate an output feature map. A representation of the network is generated as a linear sequence of operators. Operators are identified in the sequence that cannot form part of a cascade and are to be processed with the entire input and output feature map of the respective operator in the storage. The method forms one or more cascades of two or more successive operators in the sequence for which the input feature map of each operator of the cascade is processed in portions, which portions are less than the entire input feature map. The method forms the one or more cascades by sequentially, from one end of the sequence of operators, designating each operator that could form part of a cascade as a member of a cascade or as an operator to be processed with the entire input and output feature map of the operator in the storage.