Accelerator Scheduler for Multi-Application Priority Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing systems with multiple accelerators experience performance contention and inference issues due to lack of proper access scheduling, leading to unsatisfactory user experiences, especially in multi-tenant environments where multiple applications share physical hardware resources.
Innovation Solution
A hardware-based architecture that schedules access to multiple accelerators based on priorities defined using multiple granularities, embedded in request descriptors, and applies Quality-of-Service (QoS) schemes such as service level agreements (SLAs) or service level objectives (SLOs) to optimize resource allocation and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple applications share physical hardware resources without proper access scheduling, then hardware utilization is improved, but performance contention and inference issues occur
Solution Approach 1:
A hardware scheduler is introduced as an intermediary component between applications and accelerators. The scheduler receives requests from multiple applications, manages access to shared hardware resources, and ensures fair allocation based on priority levels. This mediator prevents direct performance contention by coordinating access before it reaches the accelerator hardware.
Solution Approach 2:
The access control mechanism is segmented into multiple priority levels (e.g., real-time, non-real-time, best-effort). Each priority level is handled separately by the scheduler, allowing fine-grained control over resource allocation. This segmentation enables the system to guarantee performance for critical applications while still allowing lower-priority applications to access hardware resources.
2Productivity
If sequential or parallel access is implemented without scheduling, then accelerator utilization is improved, but performance contention increases
Solution Approach 1:
The scheduler performs preliminary actions by pre-establishing priority levels and allocation policies before actual hardware access is needed. Applications submit requests with associated priority indications, and the scheduler pre-processes these requests to determine the execution order and timing. This preliminary scheduling eliminates the need for runtime arbitration, reducing access contention time.
Solution Approach 2:
The scheduling mechanism dynamically adjusts access allocation based on real-time priority indications from applications. When an application needs higher priority access, the scheduler can dynamically reassign resources without affecting other running applications. This dynamic adaptation allows the system to maintain high accelerator utilization while minimizing time loss due to contention.
3Adaptability or versatility
If multiple applications share accelerators simultaneously, then system versatility is improved, but quality of service deteriorates
Solution Approach 1:
Different quality levels are assigned to different application requests based on their priority indications. The scheduler applies local quality control by ensuring that real-time applications receive guaranteed resource allocation with minimal latency, while non-real-time applications receive best-effort service. This local differentiation maintains high service quality for critical applications without sacrificing support for less critical ones.
Data Source
AI summary
Methods, apparatus, and computer programs are disclosed to schedule access to multiple accelerators. In one embodiment, a method is disclosed to perform: receiving a first request to process data for a first application by a first accelerator of a plurality of accelerators of a computing system, an accelerator of the plurality of accelerators being dedicated to one or more respective specialized computations of the computing system for data processing; scheduling resources for the first request based on the first request and a second request to process data for a second application by a second accelerator of the plurality of accelerators, the first and second requests having one or more priority indications indicating priority between the first and second requests; and processing the data for the first application using the resources as scheduled responsive to the first request.