Virtual Tile Cluster Middleware for GPU Task Scheduling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional GPU architectures and scheduling policies limit flexible task scheduling and resource utilization, particularly in applications requiring multi-processor synchronization, leading to underutilization of GPU resources due to bulk-synchronous programming models and lack of manual algorithm placement on multi-processors.

Innovation Solution

A software middleware that presents a GPU device as a virtual compute cluster, forming a virtual tile cluster with tiles comprising processors and memory from both CPU and GPU devices, and a virtual unified memory with ingress and egress rings, allowing for efficient task assignment, execution, and data management across subprocessors, enabling manual placement and synchronization of tasks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional bulk-synchronous programming model is used on GPU, then computational throughput is maximized, but task scheduling flexibility is lost

Engineering Contradiction:

Improvecomputational throughputVSAvoidtask scheduling flexibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent segments the GPU architecture into multiple virtual multi-processors (vMPs), each capable of independent task scheduling and execution. This segmentation allows different task scheduling policies to be applied to different vMPs simultaneously, maintaining high throughput while enabling flexible task scheduling for specific algorithms.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a software layer that acts as an intermediary between the host CPU and the GPU hardware, implementing a virtualized scheduling mechanism. This software intermediary enables manual algorithm placement and task scheduling on specific vMPs while maintaining compatibility with the underlying bulk-synchronous execution model.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Device complexity

If hardware-defined scheduling policies are used, then implementation simplicity is maintained, but manual algorithm placement on specific multi-processors is prevented

Engineering Contradiction:

Improvescheduling implementation complexityVSAvoidmanual algorithm placement capability

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If GPU resources are allocated for massive parallel processing, then arithmetic operation throughput increases, but resource utilization for flexible applications decreases

Engineering Contradiction:

Improvearithmetic operation throughputVSAvoidresource utilization for flexible applications

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies different scheduling policies and resource allocation strategies to different virtual multi-processors based on their specific workload requirements. This local quality approach allows optimal resource utilization for each type of application while maintaining overall system throughput.

Inventive Principle:

Principle #3Local quality

4Adaptability or versatility

If multi-processor synchronization is implemented, then task coordination capability is improved, but risk of deadlock increases

Engineering Contradiction:

Improvetask coordination capabilityVSAvoiddeadlock risk

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent implements a synchronization mechanism with feedback control that monitors the state of multi-processor operations and dynamically adjusts scheduling decisions. This feedback mechanism detects potential deadlock conditions and takes corrective action, enabling safe multi-processor synchronization for coordinated task execution.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10963300B2Accelerating dataflow signal processing applications across heterogeneous CPU/GPU systems

Publication Date: 2021.03.30 RAYTHEON CO

AI summary

A method includes: forming a virtual tile cluster having tiles, wherein a tile comprises a processor and memory from a CPU device and a GPU device, and a tile in the GPU device further comprises subprocessors; forming a virtual unified memory that is accessible by the CPU and GPU devices; receiving a task; assigning the task to a tile of the virtual tile cluster according to a pre-defined rule. When the task is assigned to a tile in the GPU device, the method further performs: broadcasting the task to the subprocessors of a tile using a GPU shuffle instruction; and dividing data for the task and assigning the divided data to the subprocessors, wherein each subprocessor runs a codelet using the each of divided data. The task is executed by the at least one tile of the virtual tile cluster.