Device Prefetching Logic for Offloaded Data Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Disaggregated data centers face service latency issues when offloading data processing from a host processor to an accelerator device due to high cache miss rates and memory page faults, particularly in services like compression/decompression, where data is processed in 'one-shot' mode and memory page faults introduce additional latency.
Innovation Solution
Implementing prefetching logic and circuitry at the device to proactively cache data and address translation entries from the host processor's system memory, allowing for parallel data processing and reducing cache miss rates to near zero by predicting and prefetching data expected for the offloaded workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data processing is offloaded from host processor to accelerator device, then processing capability is improved, but service latency increases due to data movement overhead
Solution Approach 1:
The system performs preliminary actions by prefetching data from host memory to device memory before the actual processing workload arrives. This advance data preparation eliminates waiting time during processing, as the data is already available in device memory when needed, thus reducing service latency while maintaining improved processing capability through offloading.
2Speed
If cache is used to reduce data access latency, then data access speed is improved, but cache miss rate increases in one-shot processing mode
Solution Approach 1:
The system prefetches data into device memory in advance of the processing workload, ensuring that when one-shot processing occurs, the data is already present in device memory rather than requiring cache retrieval. This preliminary data preparation eliminates cache misses for one-shot processing scenarios, maintaining both fast data access and high cache hit rates.
Data Source
AI summary
Techniques to reduce data processing latency for a device. Circuitry at a device coupled with a host processor can facilitate execution of parallel tasks associated with processing data for a service offloaded to the device from the host processor. The parallel tasks can include prefetching information for address translations related to a shared virtual memory (SVM) space that is shared between the device and the host processor and prefetching data to be processed by device in relation to the offloaded service.