ML Accelerator Preemption Checkpoints for Low-Latency Task Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning hardware accelerators struggle with efficiently handling long-running low-priority processes while ensuring low-latency execution of higher-priority tasks, leading to inefficiencies and delays in task completion.
Innovation Solution
A machine learning hardware accelerator system that includes a scalar core and ASICs, which implements preemption checkpoints and context switching to manage resource allocation and execution of higher-priority processes by saving and restoring the state of long-running processes, allowing for efficient context switching and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the hardware accelerator executes long-running low-priority processes continuously, then resource utilization is maintained, but higher-priority tasks experience delays and increased latency
Solution Approach 1:
The system dynamically switches between executing long-running low-priority processes and high-priority tasks based on preemption checkpoints and resource availability. The hardware accelerator can transition from a steady state of executing low-priority work to an interrupted state when high-priority tasks arrive, optimizing response time while maintaining overall resource utilization through conditional preemption rather than static scheduling
Solution Approach 2:
Preemption checkpoints are inserted at compile-time into the low-priority process code at strategically chosen locations where saving and restoring context has minimal impact. This preliminary preparation enables rapid preemption when high-priority tasks arrive, reducing the latency penalty by having save/restore points already in place rather than creating them dynamically during preemption events
2Productivity
If the system implements preemption and context switching, then higher-priority tasks can be executed timely, but system complexity increases due to save/restore operations
Solution Approach 1:
The preemption management is segmented into distinct components: compile-time insertion of preemption checkpoints, runtime detection of high-priority task arrival, resource availability checking, and context save/restore operations. This segmentation allows each component to be optimized independently and reduces overall system complexity by organizing the preemption mechanism into manageable, modular parts with clear interfaces
Solution Approach 2:
The hardware accelerator includes dedicated circuitry and logic that automatically perform context saving and restoring operations when preemption occurs. Rather than requiring software to manually manage context switching, the system self-services by having hardware automatically handle the save/restore of register states, memory pointers, and other context information, reducing the operational complexity burden on software
3Loss of time
If the accelerator pauses long-running processes to accommodate high-priority tasks, then latency requirements are met, but resource utilization decreases
Solution Approach 1:
The system changes the operational parameters of low-priority processes by pausing execution at preemption checkpoints rather than terminating them. This parameter change allows the same computational resources to be temporarily reallocated to high-priority tasks while preserving the low-priority process state for later resumption, thereby meeting latency requirements without permanently losing resource productivity
Solution Approach 2:
When preemption occurs, the system temporarily discards the current execution state of low-priority processes by saving it to a data structure and clearing compute unit memory. After high-priority tasks complete, the system recovers the discarded state by restoring saved values and resuming execution, ensuring that resource utilization is maintained over the longer term while accommodating urgent tasks in the short term
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure relates to a method, system and non-transitory computer readable storage medium. The method involves operating a machine learning accelerator, comprising steps of: executing, by a scalar core directing a plurality of compute units of the machine learning accelerator, a first process in a first context, wherein the first process is a long-running process; identifying, by a job scheduler of the machine learning accelerator, that a second process is queued, wherein the second process has a higher priority than a priority of the long-running process, and upon reaching a preemption checkpoint: determining, by the scalar core, an amount of available resources and in response to the amount of available resources being sufficient for the second process: pausing, by the scalar core, execution of the first process; allocating available resources to the second process; switching, by the scalar core, to a second context; executing, by the scalar core, the second process; switching, by the scalar core, to the first context; and resuming, by the scalar core, execution of the first process.