Non-disruptive Firmware Upgrade Symmetric Hardware Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Firmware upgrades for hardware accelerators, such as Intel QuickAssist Technology (QAT) devices, disrupt computing platforms by rendering hardware accelerators unavailable during the upgrade process, leading to performance degradation, delays, and latencies.
Innovation Solution
An adaptive non-disruptive firmware upgrade method that utilizes CPU fallback and workload rebalancing to redirect tasks during the upgrade, minimizing CPU burden by prioritizing CPU cost-effective workloads and rebalancing workloads across hardware accelerators based on CPU utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If firmware upgrade is performed on hardware accelerator, then firmware is updated to improve functionality and performance, but hardware accelerator becomes unavailable causing service disruption and performance degradation
Solution Approach 1:
The system segments the hardware accelerator pool into multiple groups, allowing firmware upgrades to be performed on one segment while other segments continue to handle workloads. This segmentation enables progressive upgrading without complete service interruption.
Solution Approach 2:
The system introduces a workload management intermediary that dynamically redirects workloads between hardware accelerators during firmware upgrades. This mediator ensures continuous service availability by routing traffic to available accelerators while upgrade in progress.
2Ease of manufacture
If hardware accelerator is taken offline for firmware upgrade, then firmware can be updated without interference, but task processing capacity is reduced and delays occur
Solution Approach 1:
The system performs preliminary workload redistribution before firmware upgrade begins, preparing the system to handle the reduced capacity. Workloads are pre-scheduled and routed around the upgrading accelerator to minimize delays.
Solution Approach 2:
The system maintains continuous task processing by dynamically allocating workloads to available hardware accelerators during the firmware upgrade, ensuring that useful action continues without interruption despite reduced capacity.
3Duration of action of moving object
If multiple hardware accelerators are upgraded simultaneously, then upgrade is completed faster, but system complexity and coordination difficulty increase
Solution Approach 1:
The system divides the hardware accelerator fleet into manageable segments or groups, allowing coordinated simultaneous upgrades within each segment while maintaining overall system control. This reduces coordination complexity compared to managing all accelerators as a single group.
Solution Approach 2:
The system employs dynamic workload management that automatically adapts to the state of upgrading accelerators, dynamically adjusting task distribution to maintain optimal performance while multiple accelerators are being upgraded simultaneously.
Data Source
AI summary
In a symmetric hardware accelerator system, an initial hardware accelerator is selected for an upgrade of firmware. The initial and other hardware accelerators handle workloads that have been balanced across the hardware accelerators. Workloads are rebalanced by directing workloads having low CPU utilization to the initial hardware accelerator. A CPU fallback is conducted of the workloads of the initial hardware accelerator to the CPU. While the CPU is handling the workloads, firmware of the initial hardware accelerator is upgraded.