Tensor Memory Allocation Planning for Parallel Neural Network Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and inferencing with neural networks require significant memory, time, and computing resources, which can be optimized for more efficient use.
Innovation Solution
A system for generating a memory allocation plan that includes a tensor memory allocator and stream scheduler to optimize memory usage by allocating tensors efficiently, allowing for parallel execution of operations and reducing runtime overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional memory allocation is used for neural networks, then memory management is simple, but total memory required increases significantly
Solution Approach 1:
The system performs live interval analysis and memory allocation planning before executing neural network operations. By determining which tensors are alive during each operation and pre-planning memory allocations based on this analysis, the system optimizes memory usage without requiring complex runtime memory management, thus reducing total memory required while maintaining manageable complexity
Solution Approach 2:
The system changes the parameter of memory allocation from traditional static or simple dynamic allocation to a live interval-based allocation strategy. By analyzing the temporal lifecycle of each tensor and allocating memory based on these intervals, the system achieves more efficient memory utilization, reducing the total memory footprint while introducing sophisticated allocation logic
2Reliability
If larger batch sizes are used for neural network training, then training accuracy improves, but memory requirements increase
Solution Approach 1:
The system performs preliminary live interval analysis to determine when tensors are needed during training operations. By planning memory allocations based on these intervals, the system enables larger batch sizes to improve training accuracy while managing memory requirements through optimized allocation timing, allowing the system to achieve better accuracy without proportional increases in memory
Solution Approach 2:
The system dynamically adjusts memory allocation based on the operational context and tensor lifecycles. By making memory allocation adaptive rather than static, the system can support larger batch sizes for improved training accuracy while optimizing memory usage according to actual runtime needs, thus achieving both higher accuracy and efficient memory utilization
3Quantity of substance
If memory allocation is optimized for each tensor individually, then memory efficiency improves, but allocation time increases
Solution Approach 1:
The system performs memory allocation planning as a preliminary step before execution. By analyzing live intervals and determining optimal memory allocations in advance, the system achieves high memory efficiency without incurring significant allocation time during runtime, as the planning phase completes before the actual tensor operations begin
Solution Approach 2:
The system changes the allocation approach from runtime individual tensor allocation to a pre-computed plan-based allocation. By transforming the allocation problem into a planning phase that analyzes tensor lifecycles and produces optimized allocation maps, the system achieves superior memory efficiency while keeping allocation time minimal, as the heavy computation occurs before execution rather than during it
Data Source
AI summary
Apparatuses, systems, and techniques to generate a memory allocation plan for a set of tensors. In at least one embodiment, tensor data corresponding to the set of tensors is stored into memory locations at run time based, at least in part, on the memory allocation plan generated at a compile time.