Neural Network Memory Allocation Using Heap Free Areas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Integrated circuits face challenges in optimizing the storage of intermediate results and scratch data due to limited memory size, leading to inefficiencies in RAM and flash memory usage and increased inference time.
Innovation Solution
A method is proposed for determining the execution order of neural network layers and allocating intermediate result buffers and scratch buffers in a volatile memory's heap zone based on this order, utilizing free areas to optimize memory usage without increasing RAM or flash memory size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If scratch data is stored in RAM or flash memory, then inference time is reduced, but memory size requirements increase
Solution Approach 1:
The patent implements dynamic memory allocation where scratch buffers are allocated from the heap memory zone during layer execution based on available free space. The allocation is adaptive - buffers are created when needed and released when no longer required, allowing the system to optimize inference time without committing to fixed large memory allocations. This dynamic approach enables scratch data to reside in faster memory (RAM heap) rather than slower flash memory.
Solution Approach 2:
The patent employs a buffer release mechanism where intermediate result buffers and scratch buffers are discarded and their memory space recovered back to the heap when no longer needed for current layer execution. This allows the same memory space to be reused for different purposes at different times, reducing the total memory footprint while maintaining fast access for active scratch data.
2Productivity
If intermediate result buffers are allocated in heap memory zone, then scratch buffer allocation is optimized, but memory management complexity increases
Solution Approach 1:
The patent makes the heap memory zone serve multiple functions: it acts as both the allocation pool for intermediate result buffers and the source of free space for scratch buffer allocation. By defining scratch buffer placements in the free areas of the same heap memory zone used for intermediate results, the system eliminates the need for separate scratch memory areas, simplifying overall memory architecture despite the dynamic management requirements.
Solution Approach 2:
The patent performs preliminary determination of heap memory zone boundaries and free areas before scratch buffer allocation. By pre-establishing the heap zone and tracking its free space based on intermediate result buffer placements, the system prepares the memory landscape in advance, making scratch buffer allocation more efficient and reducing runtime management complexity.
Data Source
AI summary
According to an aspect, a method is proposed for defining placements, in a volatile memory, of temporary scratch buffers used during an execution of an artificial neural network, the method comprising: determining an execution order of layers of the neural network, defining placements, in a heap memory zone of the volatile memory, of intermediate result buffers generated by each layer, according to the execution order of the layers, determining at least one free area of the heap memory zone over the execution of the layers, defining placements of temporary scratch buffers in the at least one free area of the heap memory zone according to the execution order of the layers.