Neural Network Depth-First Processing Cache Thrashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face high computational requirements and cache thrashing issues due to the generation of large amounts of temporary data during breadth-first processing, leading to inefficient use of CPU and GPU caches and prolonged training and prediction times.
Innovation Solution
Implementing a depth-first processing method where neural network layers meeting a locality constraint are identified, and code is generated to perform depth-first processing, allowing for computation of sub-regions of input data before moving on to the next, thereby reducing the amount of temporary data that needs to be stored in caches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If breadth-first processing is used to evaluate neural networks, then computational completeness is achieved, but cache efficiency deteriorates due to large amounts of temporary data generation
Solution Approach 1:
The patent inverts the conventional breadth-first processing order by implementing depth-first processing. Instead of computing all neurons in layer 1 before moving to layer 2, the method computes neurons in a depth-first traversal order (e.g., layer 1 neuron 1, then layer 2 neuron 1, then layer 3 neuron 1, etc.). This inversion allows intermediate results to be reused across layers without regenerating them, significantly reducing temporary data generation and improving cache efficiency while maintaining computational completeness.
2Ease of operation
If conventional breadth-first methods are used, then implementation simplicity is maintained, but processing speed decreases due to frequent main memory fetches
Solution Approach 1:
The patent introduces dynamic code generation that adapts the processing order to the specific neural network architecture. The system analyzes the network structure and generates optimized depth-first traversal code that dynamically adjusts the computation sequence. This dynamic approach maintains ease of operation through automated code generation while dramatically improving processing speed by keeping data in cache memory rather than requiring frequent main memory fetches.
Data Source
AI summary
A method for accelerating a neural network includes identifying neural network layers that meet a locality constraint. Code is generated to implement depth-first processing for different hardware based on the identified neural network layers. The generated code is used to perform the depth-first processing on the neural network based on the generated code.