AI Processor Memory Allocation for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current memory allocation methods for AI processors in neural network inference processes are inefficient, leading to high memory consumption and reduced execution efficiency due to inadequate allocation strategies that do not account for the specific memory needs of each operator in the neural network.
Innovation Solution
A hybrid memory allocation method combining static and dynamic memory pool allocation, where operators with fixed input sizes use static memory pool allocation and those with variable input sizes employ dynamic memory pool allocation, with heuristic algorithms to optimize memory block sizing and placement, allowing for overlap of input and output memory blocks to minimize memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If dynamic memory allocation method is used to reduce memory consumption, then memory space occupied is reduced, but memory needs to be frequently allocated and released which affects execution efficiency and increases time consumption
Solution Approach 1:
The patent segments the neural network operators into different types based on their memory characteristics (fixed input size vs. variable input size). Static memory pool allocation is applied to operators with fixed input sizes, while dynamic memory pool allocation is applied to operators with variable input sizes. This segmentation allows each operator type to use the most appropriate allocation strategy, reducing overall memory consumption without significantly impacting execution efficiency.
2Quantity of substance
If static memory pool allocation method is used to reduce memory consumption, then memory space is uniformly allocated and released, but memory is not properly allocated based on specific situation of each operator resulting in high memory usage
Solution Approach 1:
The patent applies different memory allocation strategies to different operators based on their local characteristics. Operators with fixed input sizes use static memory pool allocation, while operators with variable input sizes use dynamic memory pool allocation. This local differentiation ensures that each operator receives the most appropriate allocation method for its specific needs, improving both memory efficiency and allocation adaptability.
Solution Approach 2:
The patent introduces a dynamic component to the memory allocation system by implementing a dynamic memory pool allocation method for operators with variable input sizes. This allows the memory allocation strategy to adapt dynamically to the specific requirements of different operators, rather than using a uniform static approach for all operators.
3Quantity of substance
If in-place processing is used to reduce memory consumed by operators, then memory consumption is reduced, but technical efficiency of the neural network is affected
Solution Approach 1:
The patent segments operators based on their input size characteristics and applies different memory allocation strategies accordingly. By identifying operators with fixed input sizes and applying static memory pool allocation to them, the patent achieves memory reduction without the performance penalties associated with in-place processing, as static allocation provides both memory efficiency and performance benefits.
Data Source
AI summary
A memory allocation method for an AI processor, a computer apparatus, and a computer-readable storage medium. The method includes: obtaining a plurality of operators of a neural network; analyzing an operator whose input and output occupy memory space that can overlap; determining whether a size of an input of the neural network is fixed; and if yes, determining storage addresses of a memory blocks by using a static memory pool allocation algorithm: calculating a size of each memory block in an inference process of a neural network model, determining a life cycle of each memory block, determining whether the memory block is a memory block that can be overlapped and if yes, correcting the size and the life cycle of the memory block, and allocating a storage address to each memory block.