Hardware-Software Coherent Framework for Accelerator Memory Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional I/O models for accelerator-memory interactions in computing systems face inefficiencies due to hardware-enforced coherency, where all interactions must go through the host CPU, leading to high latency, bandwidth, and protocol messaging overhead, especially when dealing with large data footprints, and accelerators are unable to cache memory.
Innovation Solution
A collaborative hardware-software coherent framework is introduced, where the accelerator device shares the same coherent domain as the host, allowing direct communication and management of memory, enabling software coherency to transfer ownership of data sets between the host and accelerator, reducing overhead and enabling local caching and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hardware-enforced coherency is used where all accelerator-memory interactions go through the home agent node in the main CPU-memory complex, then coherency is maintained, but latency and bandwidth overhead increase significantly
Solution Approach 1:
The patent introduces a coherency manager as an intermediary component that handles coherency tracking and management. Instead of requiring every accelerator-memory interaction to go through the home agent node, the coherency manager tracks accelerator memory accesses and manages coherency protocols, allowing direct accelerator-to-memory paths while maintaining coherency. This intermediary approach resolves the contradiction by enabling both coherency maintenance and reduced latency.
2Reliability
If all accelerator-memory interactions are serviced and tracked by the home agent node, then coherency is ensured, but protocol messaging overhead increases
Solution Approach 1:
The patent extracts the coherency tracking and management functions from the home agent node and consolidates them into a dedicated coherency manager component. This extraction allows the home agent node to focus on its primary functions while the coherency manager handles accelerator-specific coherency protocols, reducing protocol messaging overhead and simplifying the overall system architecture.
Solution Approach 2:
The coherency manager serves as an intermediary between the accelerator and the memory system, handling coherency tracking and protocol management. This intermediary approach reduces the protocol messaging overhead by providing a specialized component that understands accelerator memory access patterns and can manage coherency more efficiently than the general-purpose home agent node.
3Adaptability or versatility
If accelerators use remote server host attached memory, then memory access is possible, but bandwidth and latency performance deteriorate
Solution Approach 1:
The patent segments the memory hierarchy into local accelerator memory and remote host memory, with the accelerator having direct access to local memory for high-bandwidth operations. This segmentation allows the accelerator to use fast local memory for compute-intensive tasks while still maintaining the ability to access remote memory when needed, thus improving overall bandwidth performance without sacrificing adaptability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An accelerator device (140) shares the same coherent domain as hardware elements in a host computing device (105). A mix of hardware and software coherency reduces the overhead of managing data when large chunks of data are moved from the host into the accelerator device. An accelerator application (125) executing on the host identifies a data set (130) it wishes to transfer to the accelerator device to be processed. The accelerator application transfers ownership from a home agent (135) in the host to the accelerator device. A slave agent (155) can then take ownership of the data. As a result, any memory operation requests received from a requesting agent (145) in the accelerator device can gain access to the data set in a local memory (160) via the slave agent without the slave agent obtaining permission from the home agent in the host.