Shared Memory Allocation for Neural Network Tensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing demand for deploying neural networks on embedded devices is hindered by the high memory consumption due to the large tensors and multiple layers, leading to memory overhead issues.
Innovation Solution
A method and apparatus for allocating memory in electronic devices that involves receiving multiple frames of speech data and inputting them into a neural network model, where the data tensors share a common memory, allowing for efficient memory allocation based on the number of frames, reducing memory overhead by reusing and releasing tensors as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple data tensors are allocated separately for processing multiple frames of speech data in a neural network model, then the neural network can process multiple frames simultaneously, but the memory overhead increases significantly
Solution Approach 1:
The patent merges multiple separate data tensor allocations into a single shared memory space. Instead of allocating independent memory for each data tensor when processing multiple speech frames, the system creates one unified memory buffer that all tensors share, thereby reducing total memory overhead while maintaining the ability to process multiple frames simultaneously through proper memory management and data layout optimization.
Solution Approach 2:
The shared memory buffer serves multiple functions: it acts as storage for multiple data tensors, provides workspace for intermediate calculations, and enables efficient data reuse across different neural network layers and operations. This multi-functional design eliminates the need for dedicated memory regions for each tensor, resolving the contradiction between processing throughput and memory consumption.
2Measurement precision
If the neural network model uses large tensors with many layers to improve speech recognition accuracy, then the processing capability increases, but the memory consumption exceeds the capacity of embedded devices
Solution Approach 1:
The patent segments the memory allocation strategy by dividing the computation into stages that reuse the same memory buffer. Instead of loading all large tensors simultaneously into memory, the system processes data in segmented batches, reusing the shared memory buffer for each batch, thereby enabling high-accuracy neural network models to run on embedded devices with limited memory capacity.
Solution Approach 2:
The patent changes the memory usage pattern from spatial (allocating separate memory for each tensor) to temporal (reusing the same memory buffer across different time steps and computation stages). This dimensional shift allows the neural network to maintain large tensor operations for high accuracy while fitting within the memory constraints of embedded devices through efficient temporal reuse of memory resources.
Data Source
AI summary
The disclosure provides a method and an apparatus for allocating memory, and an electronic device. Multiple frames of speech data are received and input to a neural network model. The neural network model is configured to ask for multiple data tensors when processing the multiple frames of speech data, and the multiple data tensors share a common memory.