Neural Network Operation Decomposition for Cache Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face performance issues due to memory access inefficiencies, as operations often require storing large outputs in slower memory like DRAM, leading to slower execution on devices with limited computational resources.
Innovation Solution
A decomposition process is implemented to split neural network operations into smaller components that fit within the cache of the target device, reducing memory traffic and avoiding slower memory access, while maintaining accuracy by ensuring the aggregate output of decomposed operations matches the original operation's output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If neural network operations store large outputs in DRAM, then memory capacity is sufficient, but execution speed decreases due to slower memory access
Solution Approach 1:
The patent divides a single neural network operation that produces large output into multiple smaller operations, each producing output that fits within the cache memory. This segmentation allows intermediate results to be stored in faster cache memory rather than slower DRAM, improving execution speed while maintaining the same overall computational function.
2Speed
If neural network operations use cache memory, then execution speed improves, but memory capacity becomes insufficient for large outputs
Solution Approach 1:
By segmenting the operation into multiple smaller operations, the patent enables the use of limited cache memory capacity to store intermediate results, while the full computational output capacity is maintained through sequential processing of segmented operations.
Solution Approach 2:
The patent performs preliminary segmentation of the neural network operation before execution, organizing the computational tasks in advance to optimize memory usage patterns and ensure that intermediate results fit within cache memory, thereby preparing the system to maximize cache utilization during actual execution.
3Loss of energy
If neural network operations are decomposed into smaller components, then memory traffic to slower memory is reduced, but operation complexity increases
Solution Approach 1:
The patent applies segmentation to reduce memory traffic by keeping intermediate results in fast cache memory, accepting increased operational complexity as a necessary trade-off to achieve significant reductions in energy-consuming memory access operations to DRAM.
Data Source
AI summary
The subject technology receives a representation of a neural network (NN) model to be executed on an electronic device, the representation of the NN model including nodes corresponding to intermediate layers of the NN model. The subject technology determines, for the respective operation corresponding to each node in each respective intermediate layer of the NN model, a respective set of operations that are mathematically equivalent to the respective operation such that an aggregation of outputs of the respective set of operations is equivalent to an output of the respective operation. The subject technology generates a graph based on each respective set of operations, wherein the graph includes a set of branches, each branch includes a plurality of operations. The subject technology determines a respective order for executing each branch of the graph.