Neural Network Cache Allocation via Compiler Indicators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Limited memory resources in computing devices, particularly in mobile devices, lead to performance penalties when executing neural networks, as they often rely on slower off-chip memory, which also consumes more power.
Innovation Solution
Incorporating cache indicators during the compilation process of neural networks to prioritize the use of faster on-chip memory, allowing hardware components to allocate cache memory effectively during runtime, thereby enabling faster access and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If neural networks are executed on off-chip memory (DRAM), then larger memory capacity is available, but access speed decreases and power consumption increases
Solution Approach 1:
The patent segments memory allocation by dividing the neural network computation into multiple stages, with different layers or operations allocated to different memory resources. Early layers that benefit from fast access are placed in cache memory, while later layers with larger data requirements use DRAM, optimizing the speed-capacity tradeoff for each segment
Solution Approach 2:
The patent applies local quality by providing different memory resources to different parts of the neural network based on their specific requirements. Critical paths and layers sensitive to access latency receive high-speed cache memory allocation, while other layers access standard DRAM, creating localized optimization throughout the network execution
2Use of energy by moving object
If neural networks are executed on off-chip memory (DRAM), then sufficient memory space is available, but power consumption increases
Solution Approach 1:
The patent segments power consumption management by identifying and prioritizing power-critical operations. It allocates cache memory specifically to layers or operations that consume the most power during memory access, thereby reducing overall power consumption while maintaining sufficient memory capacity through selective caching
3Productivity
If cache memory is allocated to neural network operations, then access speed improves, but cache memory resources are consumed
Solution Approach 1:
The patent applies partial action by allocating cache memory to only the necessary portions of the neural network execution rather than the entire network. It identifies specific layers, operations, or data sets that benefit most from caching and allocates cache resources selectively to those partial components, achieving performance improvement without exhausting cache resources
Solution Approach 2:
The patent dynamically changes memory allocation parameters based on runtime conditions, including the size of the neural network layer, the expected execution time, and the available cache resources. This allows the system to adjust the degree of caching applied to different operations, optimizing the balance between productivity and resource consumption
Data Source
AI summary
The subject technology receives code corresponding to a neural network (NN) model, the code including particular operations that are performed by the NN model. The subject technology determines, among the particular operations, a set of operations that are to be allocated to a cache of the electronic device that is to execute the NN model. The subject technology generates a set of cache indicators corresponding to the determined set of operations. The subject technology compiles the code and the generated set of cache indicators to provide a compiled binary for the NN model to execute on a target device.