Accelerator Local Memory Routing for Faster Output Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies in accessing accelerator output due to additional data transfer times from device local memory to host memory, which offsets the benefits of using accelerators for processing.
Innovation Solution
Implementing a data mover and Application Programming Interface (API) commands to manage data movement directly from accelerator local memory to host memory or persistent storage, based on criteria such as data hotness, memory speed, and persistency, allowing direct access or orchestrated movement of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is transferred from accelerator local memory to host memory through additional steps, then data can be accessed by applications, but data transfer time increases and offsets accelerator benefits
Solution Approach 1:
The accelerator local memory is segmented into two distinct regions: a first region for intermediate data and a second region for final output data. This segmentation allows the system to differentiate between data that needs to be transferred to host memory and data that can remain in the accelerator, enabling selective data movement that reduces unnecessary transfer time while maintaining accessibility.
Solution Approach 2:
The host processor acts as an intermediary that receives data from the accelerator's first region and then transfers it to the second region or directly to the application. This intermediary role allows the host to coordinate data movement efficiently, avoiding redundant transfers and reducing overall data transfer time while ensuring data is available where needed.
2Adaptability or versatility
If multiple data movement steps are implemented, then data can be directed to appropriate destinations, but system complexity increases
Solution Approach 1:
Different regions within the accelerator memory are assigned different qualities or characteristics: the first region is optimized for intermediate processing and temporary storage, while the second region is optimized for final output and direct application access. This local differentiation simplifies data movement management by clearly defining the purpose of each region without requiring complex routing logic.
Solution Approach 2:
The system dynamically selects the optimal data movement path based on the current state and requirements. The host processor can choose to transfer data directly from the first region to the application, or from the first region to the second region and then to the application, depending on factors like data size, access patterns, and system state. This dynamic approach provides flexibility while avoiding fixed complex routing infrastructure.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A system is described. The system may include a host processor, a host memory connected to the host processor, and a storage device connected to the host processor. An accelerator may communicate with the host processor. The accelerator may produce an output. The accelerator may also include a local memory, which may include a first region and a second region. The first region of the local memory of the accelerator may support a first mode, and the second region of the local memory of the accelerator may support a second mode. The accelerator may store the output of the accelerator in a destination, which may include the host memory, the storage device, the first region of the local memory of the accelerator, or the second region of the local memory of the accelerator.