GPU Workflow Snapshotting for Low-Disruption Firmware Updates

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Firmware updates on GPUs in distributed environments cause significant disruptions to workflows, leading to computational delays and data center downtime, with existing methods requiring GPUs to restart from the beginning, losing previously completed workflow progress.

Innovation Solution

Coordinate GPU workflow execution with firmware updates by pausing the workflow, capturing a snapshot of the workflow content, and resuming execution from the snapshot after the update, leveraging primary GPUs to communicate updates to neighboring GPUs in parallel.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If firmware update is performed on GPU, then hardware remains up-to-date with latest software patches, but workflow execution is disrupted and computational resources are wasted due to restart from beginning

Engineering Contradiction:

Improvefirmware update completionVSAvoidworkflow execution time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by capturing a snapshot of the workflow state before the firmware update begins. This snapshot preserves the workflow's progress, parameters, and execution context, enabling the workflow to resume from the saved state rather than restarting from the beginning after the update completes.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The firmware update mechanism acts as an intermediary between the workflow execution and the hardware update process. By introducing a snapshotting mechanism that mediates between the workflow state and the firmware update operation, the system enables both the update to complete successfully and the workflow to continue without interruption.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If firmware update is performed on GPU, then hardware remains up-to-date with latest software patches, but data center downtime increases and computational delays occur

Engineering Contradiction:

Improvefirmware update completionVSAvoiddata center downtime

Core Design Contradiction:

ReliabilityVSDuration of action of stationary object

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The workflow execution maintains continuity by resuming from the captured snapshot state after the firmware update. This eliminates the need to restart the workflow from the beginning, ensuring that computational resources continue working without interruption and minimizing data center downtime during the update process.

Inventive Principle:

Principle #20Continuity of useful action

3Reliability

If firmware update is performed on GPU, then hardware remains up-to-date with latest software patches, but computational resources are wasted due to restart from beginning

Engineering Contradiction:

Improvefirmware update completionVSAvoidcomputational resource consumption

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of discarding the workflow state during the firmware update, the system recovers and preserves it through snapshotting. The captured snapshot allows the workflow to resume from its exact state before the update, preventing waste of computational resources that would otherwise be spent on redundant re-execution of workflow tasks.

Inventive Principle:

Principle #34Discarding and recovering

4Reliability

If serial firmware update is performed on each GPU, then update can be applied individually, but update time increases and scalability is limited

Engineering Contradiction:

Improvefirmware update applicationVSAvoidupdate speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The firmware update process is segmented into independent units that can be applied to different GPUs simultaneously. Each GPU can receive and apply the firmware update independently, allowing parallel processing across multiple devices without interference, thereby significantly increasing update speed and scalability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system transitions from sequential (one-dimensional) firmware updates to parallel (multi-dimensional) updates by enabling simultaneous update operations across multiple GPUs. This dimensional change in the update process allows the system to scale horizontally, applying updates to numerous devices at the same time rather than one after another.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS20250306912A1Improved firmware update with reduced impact for workflow applications

Publication Date: 2025.10.02 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Various embodiments described herein dynamically coordinate graphics processing unit (GPU) execution of a workflow with installation of a firmware update by controlling the workflow to pause execution of the workflow, capturing a snapshot of content of a workflow application associated with the workflow, and continuing execution of the workflow based on the snapshot and after an aspect of the firmware update has been installed on the GPU. To minimize the disruption to a cluster or a node, certain embodiments cause the firmware update to be pushed to primary GPUs of primary nodes. These primary GPUs then communicate the firmware update to neighboring GPUs to cause neighboring GPUs to perform the firmware update, for example, in parallel. In this manner, certain embodiments facilitate the quicker parallel execution of the firmware update across GPUs in a data center, while coordinating the execution with workflows being executed on the GPUs.