Interleaved Data Loading System for Compute Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In memory hierarchy systems, data loading from low-level to high-level memory for operations like accumulation results in inefficient compute resource usage and longer latency due to memory barriers, where compute resources are idle, and data is loaded out of order, leading to stalls and increased latency.
Innovation Solution
Loading data based on the order of operations rather than solely on row or column order, allowing for parallel execution of data manipulation and computation, and overlapping computation with data storing to reduce the number of loading operations and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is loaded from low-level memory to high-level memory before operations, then data is available for computation, but compute resources become idle during memory barriers resulting in inefficient resource usage and longer latency
Solution Approach 1:
The patent loads data in an order that anticipates computational needs, preparing data subsets in advance according to the operation sequence. This allows computation to begin immediately when data is ready, rather than waiting for all data to be loaded, thus maintaining compute resource utilization while ensuring data availability.
Solution Approach 2:
The patent implements continuous useful action by overlapping data loading with computation operations. Instead of idle periods during memory barriers, the system continuously performs useful work by executing computations on already-loaded data subsets while simultaneously loading the next data subsets, eliminating wasted compute resource time.
2Ease of manufacture
If data is loaded in row or column order, then data loading is simple and systematic, but computation must wait for complete data loading resulting in stalls and increased latency
Solution Approach 1:
The patent segments data into multiple subsets and loads them in a specific order corresponding to computational requirements. This segmentation allows computation to proceed on available data subsets while remaining subsets are still being loaded, reducing overall latency while maintaining systematic data loading through defined segmentation rules.
Solution Approach 2:
The patent performs preliminary data loading in an operation-aware order, loading data subsets before they are needed for computation. This preliminary action eliminates waiting time during computation by ensuring data is ready in advance according to the computational sequence, reducing latency without sacrificing loading simplicity.
3Reliability
If memory barriers are inserted to ensure data visibility, then data loading is guaranteed before operations, but compute resources are forced to wait resulting in inefficient resource usage
Solution Approach 1:
The patent loads data subsets in advance according to the operation sequence, performing the data loading action before computation begins. This preliminary loading eliminates the need for compute resources to wait at memory barriers, as data is already prepared and visible when computation starts, thus maintaining reliability while reducing idle time.
Solution Approach 2:
The patent maintains continuous useful action by eliminating idle wait periods at memory barriers. Computation proceeds continuously on available data subsets while data loading continues in parallel, ensuring data visibility is maintained without forcing compute resources into idle states, thereby reducing overall execution time.
Data Source
AI summary
Systems, apparatuses and methods include technology that identifies that a computation will be executed based on a plurality of values. The technology determines an order-of-operations associated with the computation and loads the plurality of values in an order determined based on the order-of-operations.