Neural Network Memory Allocation with Reusable Heap Scratch Buffers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Integrated circuits face challenges in optimizing the storage of intermediate results and scratch data due to limited memory size, leading to inefficiencies in RAM and flash memory usage, which affects inference time and power consumption.
Innovation Solution
A method for allocating temporary scratch buffers in a volatile memory by determining the execution order of neural network layers and utilizing free areas in the heap memory zone to optimize memory usage, allowing efficient placement of intermediate result and scratch buffers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If scratch data is stored temporarily in memory during layer execution, then inference time is reduced, but memory size requirements increase
Solution Approach 1:
The patent implements dynamic memory allocation where scratch buffers are allocated from the heap memory zone during layer execution and automatically freed when the layer completes. This dynamic approach allows memory to be reused across different layers, reducing peak memory requirements while maintaining fast access during critical computation phases.
Solution Approach 2:
The patent recovers memory by automatically freeing scratch buffers after each layer execution completes. The memory management system tracks buffer usage and releases heap memory back to the free pool, enabling the same memory space to serve multiple layers sequentially, thus reducing overall memory size requirements.
2Speed
If scratch buffers are allocated in dedicated memory space, then data access speed is improved, but available memory for intermediate results is reduced
Solution Approach 1:
The patent creates a universal heap memory zone that serves dual purposes: storing intermediate results from layer executions and allocating scratch buffers during layer operations. This multi-functional memory space eliminates the need for separate dedicated scratch memory, maintaining fast access speeds while preserving memory for intermediate results.
Solution Approach 2:
The heap memory zone dynamically adjusts its allocation between intermediate results and scratch buffers based on execution phase. During layer execution, memory is allocated to scratch buffers for fast access; between executions, memory is freed and available for intermediate result storage, optimizing both speed and memory availability.
3Productivity
If memory is pre-allocated for all possible intermediate results, then memory access efficiency is improved, but memory size requirements increase
Solution Approach 1:
The patent implements dynamic memory allocation within the heap memory zone, adjusting buffer sizes and positions based on actual execution needs rather than pre-allocating fixed space for all possible intermediate results. This dynamic approach maintains efficient access patterns while adapting memory usage to actual computational requirements.
Solution Approach 2:
The memory management system continuously recovers heap memory space as intermediate results are consumed by subsequent layers. By freeing memory that is no longer needed and reallocating it to active computations, the system maintains high memory access efficiency without requiring large pre-allocated memory pools.
Data Source
AI summary
According to an aspect, a method is proposed for defining placements, in a volatile memory, of temporary scratch buffers used during an execution of an artificial neural network, the method comprising: determining an execution order of layers of the neural network, defining placements, in a heap memory zone of the volatile memory, of intermediate result buffers generated by each layer, according to the execution order of the layers, determining at least one free area of the heap memory zone over the execution of the layers, defining placements of temporary scratch buffers in the at least one free area of the heap memory zone according to the execution order of the layers.