Total Order Queue for Fair Resource Scheduling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In parallel processing systems, the existing methods for managing common resource access lead to priority inversions and inefficiencies, where newer requests are serviced before older pending requests, causing older requests to be repeatedly blocked and unable to make forward progress.

Innovation Solution

A computer-implemented method that schedules common resource access requests by delaying newer requests when the resource is not available and prioritizing older requests once the resource becomes available, using a total order queue (TOQ) to manage and allocate resources fairly, avoiding priority inversions and deadlock conditions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If newer requests are allocated common resources before older pending requests, then resource utilization is improved, but priority inversion occurs and older requests are blocked indefinitely

Engineering Contradiction:

Improveresource utilizationVSAvoidrequest completion guarantee

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent applies preliminary action by checking for pending requests before allocating common resources to new requests. When a new request arrives, the system first checks if there are older pending requests waiting for the same resource. If so, the new request is delayed until the older request completes, preventing priority inversion before it can occur.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback by continuously monitoring the state of common resources and pending requests. When a common resource becomes available, the system checks which request (new or pending) should be serviced next based on arrival time, ensuring that the allocation decision reflects the current system state and maintains fairness.

Inventive Principle:

Principle #23Feedback

2Loss of time

If pending requests are serviced immediately when resources become available, then response time is improved, but pipeline latency causes newer requests to consume resources before pending requests can complete

Engineering Contradiction:

Improverequest waiting timeVSAvoidscheduling fairness

Core Design Contradiction:

Loss of timeVSReliability

Solution Approach 1:

Before servicing a pending request when a resource becomes available, the system performs a preliminary check to see if even older requests have arrived in the meantime. This preliminary action ensures that the system maintains chronological ordering and prevents newer requests from jumping ahead of older pending requests due to pipeline latency.

Inventive Principle:

Principle #10Preliminary action

3Stability of the object's composition

If requests requiring multiple common resources are placed in pending state until all resources are available, then resource allocation consistency is improved, but pipeline latency causes other requests to consume available resources before the pending request can proceed

Engineering Contradiction:

Improveresource allocation consistencyVSAvoidrequest throughput

Core Design Contradiction:

Stability of the object's compositionVSProductivity

Solution Approach 1:

When a request requiring multiple common resources is placed in pending state, the system performs preliminary actions by checking for the availability of all required resources and reserving them if possible. This prevents other requests from consuming resources that the pending request needs, while also checking for even older pending requests that may have arrived during the waiting period.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary scheduling mechanism that mediates between requests requiring multiple resources and available resources. This intermediary system maintains a pending request queue and coordinates resource allocation, ensuring that requests receive all required resources atomically while preventing resource starvation and maintaining system-wide fairness.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS9836325B2Resource management subsystem that maintains fairness and order

Publication Date: 2017.12.05 NVIDIA CORP

AI summary

One embodiment of the present disclosure sets forth an effective way to maintain fairness and order in the scheduling of common resource access requests related to replay operations. Specifically, a streaming multiprocessor (SM) includes a total order queue (TOQ) configured to schedule the access requests over one or more execution cycles. Access requests are allowed to make forward progress when needed common resources have been allocated to the request. Where multiple access requests require the same common resource, priority is given to the older access request. Access requests may be placed in a sleep state pending availability of certain common resources. Deadlock may be avoided by allowing an older access request to steal resources from a younger resource request. One advantage of the disclosed technique is that older common resource access requests are not repeatedly blocked from making forward progress by newer access requests.