Cache-Coherent Interconnect for Low-Latency Accelerator Data Staging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for loading data into accelerators, such as machine learning models, are inefficient due to multiple copy operations and bandwidth limitations, leading to increased latency and reduced concurrent processing capabilities.
Innovation Solution
Implementing a cache-coherent interconnect protocol, like Compute Express Link (CXL), to directly access and transfer data between the host processor and accelerator memory, bypassing intermediate storage and reducing the need for data copies, while utilizing a high-bandwidth bus for efficient data movement between storage tiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional PCIe-based data transfer methods are used, then data can be transferred between host and accelerator, but transfer latency is high and bandwidth is limited
Solution Approach 1:
The patent introduces a buffer memory as an intermediary component between the accelerator and host system. This buffer memory receives input data directly from the host and makes it available to the accelerator, eliminating the need for the accelerator to wait for data transfers and reducing transfer latency significantly.
Solution Approach 2:
The patent adds a temporal dimension to data transfer by implementing asynchronous data staging. Input data is prepared and staged in the buffer memory in advance of when the accelerator needs it, allowing data preparation and accelerator execution to occur in different time dimensions rather than sequentially.
2Ease of operation
If multiple copy operations are performed to move data into accelerator memory, then data can be loaded into the accelerator, but the process becomes complex and time-consuming
Solution Approach 1:
The patent merges the data staging and data transfer operations into a single unified process. The buffer memory serves as both the staging area and the source for accelerator memory, eliminating the need for separate copy operations between multiple memory spaces and simplifying the overall data loading process.
Solution Approach 2:
The patent extracts the data copying step from the traditional data loading pipeline. Instead of copying data through multiple intermediate buffers and memory spaces, the system takes out the copying operation entirely by having the host write directly to the buffer memory, which then provides data to the accelerator.
3Reliability
If the accelerator waits for input data to be loaded before executing kernels, then data accuracy is maintained, but processing latency increases
Solution Approach 1:
The patent implements preliminary action by having the host system prepare and stage input data in the buffer memory before the accelerator is ready to process it. This allows the data to be ready and waiting when the accelerator needs it, maintaining data accuracy while enabling the accelerator to execute kernels immediately without waiting.
Solution Approach 2:
The patent ensures continuity of useful action by maintaining the buffer memory as a persistent data source. The buffer memory continuously holds staged data available to the accelerator, allowing the accelerator to maintain continuous processing without interruptions or waits for data loading, thus improving throughput while preserving data integrity.
Data Source
AI summary
An accelerator is disclosed. A tier storage may store data. A circuit may process the data to produce a processed data. The accelerator may load the data from a device using a cache-coherent interconnect protocol.