Asynchronous Execution Queue for Accelerator Hardware

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Modern high performance computing clusters rely heavily on accelerators to overcome CPU limitations, but existing frameworks like PyTorch and TensorFlow require constant maintenance and synchronization, leading to computational burdens and limitations in combining devices from different vendors.

Innovation Solution

A method for providing a full asynchronous execution queue for accelerator hardware, replacing malloc operations with asynchronous ones that return unique reference pointers, allowing for decoupling from host systems and reducing synchronization needs, and using a look-up table to translate reference pointers to memory addresses.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If synchronous malloc operations are used in execution queues, then memory allocation is simple and direct, but synchronization overhead increases and computational efficiency decreases

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidsynchronization overhead

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system pre-allocates memory pools in advance and uses reference pointers that resolve to actual memory addresses through a lookup table. This preliminary preparation eliminates the need for synchronous malloc operations during execution, allowing async operations to proceed without blocking for memory allocation while maintaining proper memory management.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A lookup table acts as an intermediary between reference pointers and actual memory addresses. The async malloc operation returns a reference pointer that resolves to the real memory address through this table, decoupling the allocation request from the actual memory assignment and enabling asynchronous execution without synchronization barriers.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If framework code is constantly updated to match AI framework changes, then compatibility is maintained, but maintenance burden and computational resources increase

Engineering Contradiction:

Improveframework compatibilityVSAvoidmaintenance burden

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system uses a lookup table that copies and stores the mapping between reference pointers and memory addresses. This allows the execution queue to work with simple reference pointers while the actual memory address resolution is handled by the pre-computed lookup table, isolating the complexity of memory management from the async execution logic.

Inventive Principle:

Principle #26Copying

3Adaptability or versatility

If multiple Docker images are maintained for different devices, then device-specific optimizations are achieved, but user time and computational resources are consumed

Engineering Contradiction:

Improvedevice supportVSAvoiduser maintenance time

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The async execution queue with reference pointers and lookup tables provides a universal interface that works across different devices and AI frameworks. The system maintains a single codebase that can target multiple devices without requiring separate Docker images, as the device-specific optimizations are handled internally through the lookup table mechanism rather than requiring separate installation packages.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11593157B2Full asynchronous execution queue for accelerator hardware

Publication Date: 2023.02.28 NEC CORP

AI summary

A method for providing an asynchronous execution queue for accelerator hardware includes replacing a malloc operation in an execution queue to be sent to an accelerator with an asynchronous malloc operation that returns a unique reference pointer. Execution of the asynchronous malloc operation in the execution queue by the accelerator allocates a requested memory size and adds an entry to a look-up table accessible by the accelerator that maps the reference pointer to a corresponding memory address.