Neural Network Graph Memory Optimization via Layer Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing pruning techniques for artificial neural networks do not effectively optimize memory usage, particularly peak memory usage, in resource-constrained embedded systems, failing to consider specific memory budgets and resulting in inefficient deployment on-chip.
Innovation Solution
A method that identifies and prunes layers responsible for peak memory usage by determining tensor working sets and their memory consumption, allowing for targeted reduction of memory usage below a pre-determined threshold, while maintaining key performance indicators through importance metrics and re-training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing pruning techniques are applied to reduce DNN complexity, then computational requirements are reduced, but memory usage (particularly peak memory) is not optimized
Solution Approach 1:
The patent segments memory usage analysis by identifying specific working sets associated with each layer, allowing targeted pruning of layers causing peak memory usage. This segmentation enables selective optimization of memory consumption in critical regions without affecting the entire network uniformly.
Solution Approach 2:
The patent performs preliminary analysis of tensor working sets and memory consumption patterns before applying pruning. By pre-identifying which layers and tensors contribute to peak memory usage, the system can plan and execute targeted pruning strategies that optimize memory efficiency while maintaining model performance.
2Adaptability or versatility
If pruning is applied without specific memory budgets, then model simplification is achieved, but resource-constrained embedded systems cannot be efficiently targeted
Solution Approach 1:
The patent introduces memory budget as a configurable parameter that guides the pruning process. By adjusting memory budget parameters, the system can adapt pruning strategies to different resource-constrained embedded systems, enabling versatile deployment across various hardware platforms with different memory capacities.
Solution Approach 2:
The patent implements feedback mechanisms that monitor memory consumption during network execution and use this information to guide iterative pruning. The system analyzes actual memory usage patterns and adjusts pruning decisions accordingly, creating a closed-loop optimization process that adapts to real hardware constraints.
3Quantity of substance
If layers are pruned to reduce peak memory usage, then memory requirements are optimized, but model accuracy may deteriorate
Solution Approach 1:
The patent applies local quality by differentiating between layers based on their specific memory consumption characteristics and importance to model accuracy. Instead of uniform pruning, the system identifies and prunes only those layers with high memory usage and low impact on accuracy, preserving critical computational functions while optimizing memory efficiency.
Solution Approach 2:
The patent employs partial action by pruning only the necessary portions of the network - specifically the layers and tensors contributing to peak memory usage - rather than applying aggressive full-network pruning. This measured approach reduces memory consumption while minimizing negative impact on model accuracy through selective, conservative pruning.
Data Source
AI summary
A computer implemented method for optimizing a memory usage of an artificial neural network graph comprising a plurality of layers and a plurality of tensors comprises the following steps: for each of the plurality of layers, determining a tensor working set, wherein the tensor working set comprises tensors that consume memory with respect to the respective layer; determining whether at least one working set of the plurality of working sets requires memory usage above a pre-determined threshold; if it is determined that at least one working set of the plurality of working sets requires memory usage above the pre-determined threshold, identifying a working set of the plurality of working sets which requires memory usage above the pre-determined threshold; identifying at least one layer responsible for the memory usage above the pre-determined threshold in the identified working set; and pruning the identified at least one layer.