Sub-tensor Column Execution for Neural Network Cache Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network execution methods require significant compute power and memory access, particularly due to the need to process each layer synchronously and store intermediate values, leading to inefficient memory usage and redundant calculations.
Innovation Solution
The approach involves partitioning neural network computations into sub-tensor columns, where each sub-tensor column performs computations entirely within a cache layer and saves outputs within a higher cache or memory layer, reducing memory access and redundant processing by executing tasks asynchronously across layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural network layers are executed synchronously with all nodes in a layer completing before moving to the next layer, then computation accuracy is maintained, but memory usage and computation time increase significantly
Solution Approach 1:
The patent segments the neural network computation into independent sub-tensor columns that can be executed asynchronously. Instead of waiting for all nodes in a layer to complete, the system divides the computation into vertical columns spanning multiple layers, allowing each column to be processed independently and overlapped with other column executions, thus reducing total execution time while maintaining accuracy through proper synchronization at boundary points.
Solution Approach 2:
The patent implements preliminary actions by pre-computing and storing intermediate results in cache memory before they are needed by subsequent layers. This allows the system to prepare data in advance during idle periods, so that when computation proceeds, the data is already available, reducing waiting time and enabling asynchronous execution without sacrificing accuracy.
2Reliability
If intermediate layer values are stored in memory to support synchronous layer execution, then computation correctness is ensured, but memory bandwidth consumption and energy usage increase
Solution Approach 1:
The patent introduces cache memory as an intermediary between registers and main memory for storing intermediate layer values. This intermediary layer provides fast access to frequently used intermediate results, reducing the need for high-energy main memory accesses while maintaining computation correctness. The cache acts as a buffer that holds intermediate values needed for asynchronous column execution, eliminating energy-intensive memory bandwidth consumption.
3Quantity of substance
If the width of the largest layer determines memory size for intermediate values, then all layer outputs can be stored, but memory size requirements scale with network width rather than computation needs
Solution Approach 1:
The patent segments intermediate value storage into sub-tensor column units, where only the intermediate values needed for each specific column computation are stored in cache. This segmentation allows memory capacity to be proportional to the actual computation requirements of individual columns rather than scaling with the total network width, improving computation efficiency by eliminating storage of unnecessary intermediate values.
Solution Approach 2:
The patent applies local quality by allocating cache memory specifically for the intermediate values of active sub-tensor columns rather than uniformly allocating memory for all possible layer outputs. This localized allocation ensures that memory capacity matches the actual computation needs of each column, improving productivity by reducing both memory size requirements and access latency for relevant data.
4Reliability
If computations are memory-bound due to frequent memory access, then data availability is ensured, but compute unit utilization and processing speed decrease
Solution Approach 1:
The patent implements preliminary action by pre-loading intermediate values into cache memory before they are needed for computation. This allows compute units to access data from fast cache rather than waiting for slow memory accesses, transforming memory-bound computations into compute-bound operations. Data availability is maintained through careful cache management, while processing speed improves by eliminating memory access bottlenecks.
Solution Approach 2:
The patent enables continuity of useful action by keeping intermediate values in cache memory throughout the asynchronous column execution process, allowing compute units to continuously process data without interruption from memory accesses. This continuous computation maintains data availability while maximizing processing speed by ensuring compute units are always working with readily available data from cache.
Data Source
AI summary
Embodiments of the invention may execute a NN by executing sub-tensor columns, each sub-tensor column including computations from portions of a layers of the NN, and each sub-tensor column performing computations entirely within a first layer of cache (e.g. L2 in one embodiment) and saving its output entirely within a second layer of cache (e.g. L3 in one embodiment). Embodiments may include partitioning the execution of a NN by partitioning the execution of the NN into sub-tensor columns, each sub-tensor column including computations from portions of layers of the NN, each sub-tensor column performing computations entirely within a first layer of cache and saving its output entirely within a second layer of cache.