Nested Loop Reordering for Data Locality in Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks consume extensive resources, making it challenging to utilize them on real-time, energy-sensitive applications such as low-power IoT devices and systems like CPUs, DSPs, and MCUs due to high memory access costs and poor data locality, which leads to increased power consumption and cache misses.
Innovation Solution
A method is provided to optimize nested loops in program code by determining an improved loop order that aligns input and output data layouts, transforming loops using loop unrolling and scalar register replacement, and generating new program code to enhance data locality and reduce memory access, thereby improving execution speed and reducing energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks are implemented on resource-constrained systems, then classification performance is improved, but power consumption and memory access costs increase
Solution Approach 1:
The patent transforms the computational parameters by changing the loop execution order and data access patterns. Specifically, it reorders nested loops to access memory in a more sequential and localized manner, changing how computations are performed without altering the underlying neural network model. This parameter transformation reduces random memory accesses and improves cache utilization, thereby lowering power consumption while maintaining classification accuracy.
Solution Approach 2:
The patent applies local optimization to specific regions of the computation by optimizing individual nested loop structures. Each loop transformation is tailored to the local data access patterns in that particular computation block, improving data locality where it matters most. This localized approach to optimization allows the system to achieve better power efficiency in critical computation regions without requiring a complete system redesign.
2Reliability
If deep neural networks are implemented on resource-constrained systems, then classification performance is improved, but memory access costs increase
Solution Approach 1:
The patent performs preliminary actions by pre-ordering the loop structures before execution. The loop reordering is done in advance during compilation or code generation, preparing the computation in a way that minimizes future memory access costs. By anticipating and pre-optimizing the access patterns, the system reduces the energy cost of memory operations during actual neural network inference without compromising the classification performance.
3Ease of manufacture
If traditional loop structures are used in program code, then implementation simplicity is maintained, but data locality and execution speed deteriorate
Solution Approach 1:
The patent introduces dynamic characteristics to the loop structures by making the loop order adaptive to the data access patterns. Rather than using a fixed, simple loop structure, the system dynamically determines the optimal loop ordering based on the specific computational requirements and data layouts. This dynamic approach allows the code to automatically optimize for execution speed in different contexts while maintaining reasonable implementation complexity through systematic transformation rules.
4Ease of manufacture
If traditional loop structures are used in program code, then implementation simplicity is maintained, but data locality deteriorates
Solution Approach 1:
The patent segments the computation into distinct loop blocks and optimizes each segment independently. By dividing the nested loop structure into manageable segments, the system can apply specific reordering strategies to each segment to improve data locality. This segmentation approach maintains implementation simplicity by breaking down a complex optimization problem into smaller, more tractable pieces that can be systematically transformed.
Data Source
AI summary
A method receives a first program code including one or more nested loops. A loop order is determined for the nested loop(s). The determined loop order aligns an input data layout and an output data layout. The nested loop(s) are transformed based on the loop order. A second program code is generated based on the transformed nested loop(s).