Neural Network Internal Memory Caching for Data Transfer Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network systems face inefficiencies in operation processing due to high memory usage and data transfer between internal and external memory, which hinders the performance of deep learning tasks.
Innovation Solution
A neural network system that utilizes an operation processing graph with an internal memory to store and reuse feature maps, allowing for efficient computation by performing operations sequentially within the internal memory before transferring results to external memory, thereby reducing data input/output operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If feature maps are stored in external memory and accessed during operation processing, then memory capacity is sufficient, but data transfer between internal and external memory increases processing time and reduces operation speed
Solution Approach 1:
The patent implements a nested memory structure where internal memory is embedded within the operation processor, creating a hierarchical memory system. Feature maps are first cached in the internal memory (inner layer), and only data requiring further processing is transferred to external memory (outer layer). This nested arrangement reduces the frequency and volume of data transfers between memory levels, thereby improving operation processing speed while minimizing time loss.
Solution Approach 2:
The patent performs preliminary action by pre-loading feature maps into internal memory before operation processing begins. The system identifies frequently accessed feature maps and stores them in the internal memory in advance, so that when operations need to access these feature maps, the data is already available in the fast internal memory rather than requiring retrieval from external memory during processing, thus reducing data transfer time and improving speed.
2Productivity
If feature maps are repeatedly transferred between internal and external memory, then memory capacity requirements are met, but the number of data input/output operations increases and reduces processing efficiency
Solution Approach 1:
The patent ensures continuity of useful action by maintaining feature maps in internal memory throughout the operation processing sequence. Once feature maps are loaded into internal memory, they remain there for the duration of all operations that need to access them, eliminating repeated data I/O operations. This continuous availability of data in the internal memory allows processing to proceed without interruption from data transfer operations, thereby improving productivity while reducing the quantity of data I/O operations.
Solution Approach 2:
The patent creates copies of feature maps in the internal memory, separate from the original storage in external memory. These copies allow the operation processor to access feature map data multiple times without requiring repeated reads from external memory. The copying mechanism enables the system to satisfy memory capacity requirements by utilizing the internal memory buffer, while simultaneously reducing the number of data I/O operations by serving multiple read requests from the copied data rather than repeatedly accessing external memory.
3Speed
If internal memory is used to store feature maps for multiple operations, then operation processing speed improves, but internal memory usage increases
Solution Approach 1:
The patent applies local quality by selectively storing only those feature maps in internal memory that are frequently accessed or require rapid access during operation processing. Not all feature maps are loaded into internal memory; instead, the system identifies and prioritizes critical feature maps based on their access patterns and operational importance. This selective approach allows the system to improve processing speed for operations requiring fast access while limiting internal memory usage to only the necessary subset of feature maps, rather than loading all feature maps into internal memory.
Solution Approach 2:
The patent dynamically adjusts the parameter of internal memory allocation based on operational requirements. The system monitors access patterns, operation types, and memory utilization, and adjusts the amount of internal memory dedicated to storing feature maps accordingly. When operations require fast access to feature maps, the system increases internal memory allocation; when memory pressure is high or access patterns change, the system reduces allocation or evicts less frequently accessed feature maps. This dynamic parameter adjustment allows the system to optimize the balance between processing speed and memory usage.
Data Source
AI summary
A neural network system for processing a neural network model including an operation processing graph that includes a plurality of operations, includes an operation processor including an internal memory storing a first module input feature map. The operation processor is configured to obtain a first branch output feature map by performing a first operation among the plurality of operations, based on the stored first module input feature map, and obtain a second branch output feature map by performing a second operation among the plurality of operations after the first operation is performed, based on the stored first module input feature map. The internal memory maintains storage of the first module input feature map while the first operation is performed.