Static Memory Allocation for Neural Network Tensor Activations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional machine learning frameworks use dynamic memory allocation for neural networks, leading to memory fragmentation and performance slowdowns due to inefficient allocation of memory for tensor activations during execution.
Innovation Solution
Implementing static memory allocation during the compilation phase of neural networks, where memory for tensor activations is allocated based on their sizes and assigned to memory blocks before execution, reducing fragmentation and processing requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If dynamic memory allocation is used for tensor activations during neural network execution, then the system can flexibly allocate memory as needed, but memory fragmentation occurs and processing speed decreases
Solution Approach 1:
The patent applies preliminary action by allocating memory for all tensor activations during the compilation phase before the neural network is executed. The system analyzes the computational graph, determines the memory requirements of each activation, and assigns memory blocks in advance. This eliminates the need for dynamic memory allocation during execution, preventing memory fragmentation and improving processing speed while maintaining flexibility through the upfront planning stage.
2Ease of operation
If dynamic memory allocation is used for tensor activations, then memory can be allocated during execution, but memory fragmentation occurs preventing efficient allocation of future activations
Solution Approach 1:
The system performs memory allocation during the compilation phase before execution begins. It analyzes the computational graph to determine the memory requirements of each activation and assigns memory blocks in advance, ensuring efficient memory utilization without fragmentation. This preliminary planning maintains reliability by guaranteeing that memory is always available when needed during execution.
3Adaptability or versatility
If dynamic memory allocation is used, then memory can be allocated as needed, but large amounts of processing power are required for allocation operations
Solution Approach 1:
The patent shifts memory allocation operations from the execution phase to the compilation phase. During compilation, the system analyzes the computational graph and allocates memory for all activations in advance, eliminating the need for repeated allocation operations during execution. This reduces processing power consumption during the actual neural network inference or training, as the costly allocation decisions have already been made beforehand.
Data Source
AI summary
The disclosed computer-implemented method may include compiling a neural network, and the compiling may include organizing an interconnected set of nodes in a series of layers, and for each node in each layer, assigning an associated activation of a plurality of activations. Each activation may output a respective tensor of a plurality of tensors. The compiling may also include allocating memory for the activations by determining a respective memory size for each activation, and based on the respective memory size for each activation, assigning a memory block in the neural network to the activation. The method may also include, after the allocating the memory for the activations, accessing the memory blocks to perform the plurality of activations and thereby execute the neural network. Various other methods, systems, and computer-readable media are also disclosed.