Virtualized APD Hang Detection via Scheduler Timer Reset
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtualized accelerated processing devices, hangs can occur when virtual machines fail to complete the idling process within a timeout, leading to resource sharing inefficiencies and potential system instability.
Innovation Solution
A method is implemented to detect hangs by using a hang timer, where the virtualization scheduler requests the APD to idle and, upon timeout, performs a virtual function-function level reset, informing the hypervisor and virtual machines to re-initialize the APD, ensuring seamless context switching between virtual machines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the virtualization scheduler requests the APD to idle when a time-slice expires, then resource allocation efficiency is improved, but system stability deteriorates due to potential hangs
Solution Approach 1:
The system performs preliminary actions by setting up hang detection mechanisms (timers and state tracking) before the idling process completes. This allows the system to proactively detect hangs and trigger recovery procedures, preventing system instability while maintaining efficient resource allocation through timely context switching.
2Reliability
If the system implements hang detection with timeouts, then system stability is improved, but resource allocation efficiency deteriorates due to additional overhead
Solution Approach 1:
The APD performs self-service by autonomously tracking its own state (busy vs. idle) and triggering hang detection mechanisms without requiring continuous external polling. The hang timer and state machine operate independently, allowing the system to maintain stability through minimal overhead while preserving resource allocation efficiency.
3Reliability
If the hypervisor performs a virtual function-function level reset on the APD, then system stability is improved by recovering from hangs, but device complexity increases
Solution Approach 1:
The reset mechanism is segmented into specific virtual function-level operations rather than full device resets. This allows targeted recovery of individual hung functions while preserving other APD operations, reducing the overall complexity impact while maintaining system stability through precise, localized reset actions.
Data Source
AI summary
A technique for recovering from a hang in a virtualized accelerated processing device (“APD”) is provided. In the virtualization scheme, different virtual machines are assigned different “time-slices” in which to use the APD. When a time-slice expires, the APD stops operations for a current VM and starts operations for another VM. To stop operations on the APD, a virtualization scheduler sends a request to idle the APD. The APD responds by completing work and idling. If one or more portions of the APD do not complete this idling process before a timeout expires, then a hang occurs. In response to the hang, the virtualization scheduler informs the hypervisor that a hang has occurred. The hypervisor performs a function level reset on the APD and informs the VM that the hang has occurred. The VM responds by stopping command issue to the APD and re-initializing the APD for the function.