Accelerator Interconnect Repair Without Host Draining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In conventional accelerator networks, interconnect failures require draining the entire host machine, leading to substantial degradation of utilization and service, as all accelerator chips must be marked as unavailable and processing tasks reassigned, resulting in stranded and unusable equipment.
Innovation Solution
A system with a link health monitor and pod manager detects failures, allowing for automatic repair of interconnects without draining the host, using a pod manager to identify affected slices and generate corrective actions, and enabling hot plug support in firmware for non-disruptive repairs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire host machine is drained when an interconnect fails, then system reliability is improved by isolating the failure, but hardware utilization deteriorates substantially as all accelerator chips must be marked unavailable
Solution Approach 1:
The system segments the host machine into multiple independent slices, where each slice represents a portion of accelerator chips that can be independently managed. When an interconnect failure occurs, only the affected slice is drained rather than the entire host, allowing other slices to continue operating and maintaining hardware utilization while still isolating the failure.
2Measurement precision
If all accelerator chips are marked as unavailable during interconnect failure, then measurement precision is improved by accurately identifying failed components, but loss of time increases due to substantial degradation of service
Solution Approach 1:
The system applies local quality by precisely identifying the specific slice affected by the interconnect failure rather than treating the entire host as failed. The pod manager determines which slice contains the failed interconnect and marks only that slice as unavailable, while other slices remain operational. This localized approach maintains measurement precision for failure detection while minimizing service degradation time.
3Ease of repair
If the host machine is shut down for interconnect repair, then ease of repair is improved by preventing further damage, but duration of action increases due to host downtime
Solution Approach 1:
The system extracts the affected slice from active service through draining, isolating it for repair while leaving the rest of the host machine operational. This allows technicians to repair the interconnect without shutting down the entire host, reducing host downtime while still providing a controlled environment for safe repair of the failed component.
Data Source
AI summary
A system for repair of an accelerator architecture including a plurality of accelerator host having a plurality of accelerator elements in each host and high-speed interconnects between accelerator elements and including a link health monitor service in the host determining a status of each interconnect internal and external to the host. The link health monitor service detects link status of the interconnects and communicates this to a pod manager responsible for managing a number of interconnected hosts. The pod manager communicates with a scheduler for scheduling processing tasks to each accelerator element or group of elements. Upon detection of an interconnect problem, the pod manager flags a least common ancestor in a binary tree of elements containing elements adjacent to the failed interconnect and flags the ancestor and its parent nodes as unavailable. Pod manager further generates a corrective action repairing the problem interconnect communicating the action to a technician.