Nested Tensor Processing Loops for Lower Memory Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance computing systems struggle to efficiently process complex tasks like machine learning computations with limited computational resources and high memory consumption, particularly in portable or on-board platforms.
Innovation Solution
A method for processing tensors that involves iterating through operations in loops, managing memory allocation based on data availability, and using strategies like depth-first search to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If standard high performance computing systems are used for complex machine learning computations, then processing performance is improved, but memory consumption increases and power consumption increases
Solution Approach 1:
The patent segments tensor processing into multiple loops (first loop for parent operations, second loop for child operations) that can be executed independently when data is available. This segmentation allows the system to process tensors in smaller chunks rather than loading entire tensors into memory, reducing peak memory consumption while maintaining processing performance.
Solution Approach 2:
The patent implements preliminary action by checking data availability conditions before executing operations. The system determines whether sufficient data exists in the first tensor before calculating components of the second tensor, and only launches the second loop when conditions are met. This prevents unnecessary memory allocations and allows for more efficient memory management.
2Productivity
If standard high performance computing systems are used for complex machine learning computations, then processing performance is improved, but power consumption increases
Solution Approach 1:
By segmenting the computation into nested loops that process tensors in smaller units, the system reduces the frequency of memory access operations. Since memory access is typically the most power-consuming operation in computing systems, this segmentation directly reduces power consumption while maintaining overall processing performance.
Solution Approach 2:
The patent enables continuous processing by allowing the second loop to execute as soon as sufficient data is available in the first tensor, without waiting for the entire first loop to complete. This continuous execution reduces idle time and keeps the processing pipeline full, improving performance while avoiding the power overhead of repeatedly starting and stopping computation.
3Speed
If memory is allocated for all tensor operations in advance, then processing speed is improved, but memory consumption increases
Solution Approach 1:
The patent implements dynamic memory allocation where the second loop is launched conditionally based on data availability checks. Memory for the second tensor is allocated only when sufficient data exists in the first tensor to perform the calculation. This dynamic approach maintains processing speed by avoiding unnecessary waits while reducing memory consumption by allocating only when needed.
Solution Approach 2:
The system recovers memory resources by de-allocating components of the first tensor once they are no longer required by subsequent operations. This allows memory to be reused for other purposes, reducing peak memory consumption while maintaining the processing speed needed for complex tensor computations.
Data Source
Figure 1A~1B
Figure 2
Figure 3A
AI summary
The present disclosure describes a method for processing tensors, comprising providing a component of a first tensor in a memory unit, determining one or more first operations that use at least the component of the first tensor as input, iterating through the one or more first operations in a first loop, including for a current operation of the one or more first operations in the first loop, if there are enough data in the first tensor including the component of the first tensor for the current operation to calculate a component of a second tensor using the component of the first tensor: calculating the component of the second tensor using the current operation applied at least to the component of the first tensor, writing the component of the second tensor into the memory unit, if any part of at least the component of the first tensor is no longer required by any next operation of the first loop, de-allocating the part of at least the component of the first tensor from the memory unit, and launching and iterating through a second loop embedded in the first loop, the second loop including one or more second operations that use at least the component of the second tensor as input. Furthermore, a related device, related one or more computer readable storage media and a related computing system are described.