Alternative Loop Bounds for Multi-Dimensional Tensor Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning computations using neural networks face inefficiencies due to limitations in accessing data from multi-dimensional tensors, particularly when the number of iterations exceeds the number of computing units, leading to suboptimal parallel processing and memory usage.
Innovation Solution
Implementing alternative loop bounds for nested loops, where the first loop bound is adjusted based on hardware properties like the number of computing units, allowing for parallel processing even when iterations are not a multiple, and substituting an alternative bound for the final iteration to optimize memory usage and reduce instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of loop iterations is increased to process more tensor data, then the computational throughput is improved, but the parallel processing efficiency deteriorates when iterations exceed the number of computing units
Solution Approach 1:
The patent divides the nested loop iterations into multiple segments, each segment's iteration count being adjusted to match the number of computing units. This segmentation allows the total computational workload to be distributed across multiple computing units in a structured way, maintaining parallel processing efficiency while increasing overall throughput by processing data in manageable chunks.
2Device complexity
If the loop bound is set to match the number of computing units for optimal parallel processing, then the parallel processing efficiency is improved, but the adaptability to handle varying tensor sizes deteriorates
Solution Approach 1:
The patent implements dynamic loop bound adjustment where the loop bound is not fixed but adapts based on the tensor size and the number of computing units. The system calculates appropriate loop bounds that optimize parallel processing for the current computational context, allowing the same system to efficiently handle varying tensor sizes while maintaining high parallel processing efficiency.
3Productivity
If the number of iterations of the inner loop equals the number of computing units for all iterations, then the parallel processing utilization is improved, but the memory usage and instruction overhead increase
Solution Approach 1:
The patent applies discarding and recovering by reducing loop iterations in the final iteration of the outer loop to match the remaining workload rather than maintaining the full loop bound. This discards unnecessary iterations that would not contribute to the computation, thereby reducing memory access operations and instruction overhead while maintaining high parallel processing utilization throughout the computation.
Data Source
AI summary
Methods, systems, and apparatus for accessing a N-dimensional tensor are described. In some implementations, a method includes, for each of one or more first iterations of a first nested loop, performing iterations of a second nested loop that is nested within the first nested loop until a first loop bound for the second nested loop is reached. A number of iterations of the second nested loop for the one or more first iterations of the first nested loop is limited by the first loop bound in response to the second nested loop having a total number of iterations that exceeds a value of a hardware property of the computing system. After a penultimate iteration of the first nested loop has completed, one or more iterations of the second nested loop are performed for a final iteration of the first nested loop until an alternative loop bound is reached.