Asynchronous Execution Queue for Accelerator Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern high-performance computing clusters rely heavily on accelerators to overcome CPU limitations, but existing frameworks like PyTorch and TensorFlow require constant maintenance and synchronization, leading to computational burdens and limitations in combining or mixing devices from different vendors.
Innovation Solution
A method for providing a full asynchronous execution queue for accelerator hardware, replacing malloc operations with asynchronous ones that return unique reference pointers, allowing for decoupling from host systems and reducing synchronization needs, while using a look-up table to translate reference pointers to memory addresses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous malloc operations are used in execution queues, then memory allocation can be tracked and managed, but synchronization operations increase latency and reduce computational speed
Solution Approach 1:
The patent introduces reference pointers as an intermediary mechanism between the host system and accelerator device memory. Instead of requiring synchronous malloc operations that block execution, the host generates reference pointers that the accelerator can use independently. A translation layer (lookup table) mediates between these reference pointers and actual memory addresses, enabling asynchronous operation while maintaining memory tracking reliability.
Solution Approach 2:
The host system performs preliminary actions by pre-generating reference pointers and populating the translation lookup table before the accelerator executes operations. This preliminary setup enables the accelerator to perform memory allocations and translations asynchronously without waiting for host synchronization, thereby improving computational speed while maintaining memory management reliability.
2Reliability
If frequent synchronization between host and accelerator is performed, then memory consistency can be maintained, but latency increases and computational efficiency decreases
Solution Approach 1:
The translation lookup table acts as an intermediary data structure that decouples the host and accelerator memory management. The host updates the translation table asynchronously, and the accelerator independently translates reference pointers using this table. This intermediary mechanism maintains memory consistency without requiring frequent synchronization barriers, thereby reducing synchronization latency.
Solution Approach 2:
The system enables continuous useful action by allowing the accelerator to perform computation and memory operations asynchronously using reference pointers. The host continues to update the translation table in the background without blocking accelerator execution. This continuous operation maintains memory consistency while minimizing interruptions and latency caused by synchronization.
3Productivity
If framework-specific code is maintained for each accelerator device, then device-specific optimizations can be achieved, but maintenance burden and complexity increase
Solution Approach 1:
The patent implements a universal reference pointer interface that works across different accelerator devices and frameworks. Instead of maintaining separate code paths for each device, the system uses a common reference pointer mechanism with device-specific translation tables. This universal interface reduces maintenance burden while preserving the ability to optimize for specific devices through the translation layer configuration.
Solution Approach 2:
The system segments memory management into two independent layers: the universal reference pointer generation layer (host side) and the device-specific translation layer (accelerator side). This segmentation allows the host to use a single unified approach for all devices while each accelerator maintains its own translation table for device-specific optimizations. The segmentation reduces overall system complexity by separating concerns.
4Adaptability or versatility
If multiple AI frameworks are supported with separate installations, then framework-specific functionality is preserved, but user time and computational resources are wasted
Solution Approach 1:
The reference pointer mechanism provides a universal interface that can work with multiple AI frameworks (PyTorch, TensorFlow, etc.) through a single installation. The host system generates framework-agnostic reference pointers that can be translated to device-specific memory addresses regardless of which AI framework is being used. This universality preserves framework-specific functionality while eliminating the need for separate installations and reducing user maintenance time.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A method for providing an asynchronous execution queue for accelerator hardware includes replacing a malloc operation in an execution queue to be sent to an accelerator with an asynchronous malloc operation that returns a unique reference pointer. Execution of the asynchronous malloc operation in the execution queue by the accelerator allocates a requested memory size and adds an entry to a look-up table accessible by the accelerator that maps the reference pointer to a corresponding memory address.