Deep Tensor Column Execution for Neural Network Cache Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network execution methods require significant compute power and memory access, particularly due to the need to process each layer synchronously and store intermediate values, leading to inefficiencies in memory usage and redundant calculations.
Innovation Solution
The approach involves partitioning neural network execution into sub-tensor columns, where each sub-tensor column performs computations entirely within a cache layer and saves outputs within a higher cache or memory layer, optimizing cache and compute capabilities by reducing memory access and eliminating redundant calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural network execution processes each layer synchronously and stores intermediate values in memory, then computation accuracy is maintained, but memory access and compute power requirements increase significantly
Solution Approach 1:
The patent segments the neural network execution into independent tensor column computations that can be processed asynchronously. Each tensor column is divided into smaller sub-tensor columns that can be executed independently, allowing intermediate results to be stored in cache rather than main memory, thereby reducing memory access while maintaining computational accuracy through proper result aggregation.
Solution Approach 2:
The patent introduces a new execution dimension by processing tensor columns in parallel and allowing out-of-order execution. Instead of the traditional layer-by-layer synchronous processing, the system executes computations across the tensor column dimension, enabling intermediate results to remain in cache and reducing the need for frequent memory accesses while preserving accuracy through proper synchronization mechanisms.
2Reliability
If intermediate layer values are stored in memory to support synchronous layer execution, then computation correctness is ensured, but memory size requirements are driven by the width of the largest layer
Solution Approach 1:
The patent segments the neural network into independent tensor columns that can be executed asynchronously. Each tensor column processes a subset of neurons independently, allowing intermediate results to be stored in smaller cache memories rather than requiring large main memory allocations proportional to the largest layer width. This segmentation maintains computation correctness through proper result aggregation while significantly reducing memory size requirements.
Solution Approach 2:
The patent applies local quality by allowing different tensor columns to use local cache memory independently. Each tensor column's intermediate results are stored in locally available cache rather than requiring centralized main memory, optimizing memory usage for each computational unit while ensuring overall computation correctness through coordinated result aggregation.
3Reliability
If all outputs of a prior layer are executed before feeding inputs to the next layer, then data dependencies are satisfied, but redundant calculations occur and processing time increases
Solution Approach 1:
The patent introduces dynamic execution by allowing tensor columns to be processed in any order as long as data dependencies are satisfied. Instead of rigid synchronous layer execution, the system dynamically schedules tensor column execution based on available data, enabling out-of-order processing that eliminates redundant calculations and reduces processing time while maintaining data dependency satisfaction through dependency tracking mechanisms.
Solution Approach 2:
The patent enables skipping by allowing tensor columns to be executed as soon as their input data is available, rather than waiting for all prior layer outputs to be computed. This rushing through of available computations eliminates idle waiting time and redundant calculations, reducing overall processing time while maintaining data dependency satisfaction through proper synchronization barriers.
Data Source
AI summary
Embodiments of the invention may execute a NN by executing sub-tensor columns, each sub-tensor column including computations from portions of a layers of the NN, and each sub-tensor column performing computations entirely within a first layer of cache (e.g. L2 in one embodiment) and saving its output entirely within a second layer of cache (e.g. L3 in one embodiment). Embodiments may include partitioning the execution of a NN by partitioning the execution of the NN into sub-tensor columns, each sub-tensor column including computations from portions of layers of the NN, each sub-tensor column performing computations entirely within a first layer of cache and saving its output entirely within a second layer of cache.