Non-logging Resumable Distributed Cluster Upgrade
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large distributed computing clusters, upgrading multiple machines simultaneously is inefficient and prone to failures due to the complexity of managing various updates, preserving user data, and the need for multiple reboots, while existing methods like log files are error-prone and intolerant of failures.
Innovation Solution
A distributed workflow framework that runs on each machine, autonomously managing the upgrade process through a sequence of actions, dynamically verifying completion states, and allowing for resumption from failures without redoing successful work, without relying on centralized technologies or log files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a log file is used to track completed and uncompleted actions on cluster machines, then action tracking is achieved, but the system becomes inefficient, error-prone, and intolerant of failures
Solution Approach 1:
The patent divides the centralized log file tracking system into distributed state files on each individual machine. Each machine maintains its own state file that tracks completed and uncompleted actions locally, eliminating the single point of failure inherent in centralized logging while distributing the tracking complexity across multiple independent units.
Solution Approach 2:
The patent introduces state files as intermediary objects between the upgrade process and the tracking mechanism. These state files serve as local mediators that record action states without requiring centralized log file access, enabling fault-tolerant tracking by isolating each machine's state information from potential failures in centralized systems.
2Productivity
If upgrades are performed in parallel on all cluster machines simultaneously, then upgrade time is reduced, but coordination complexity and risk of inconsistency increase
Solution Approach 1:
The patent segments the upgrade process into independent parallel workflows that run on each machine simultaneously. Each machine executes its own upgrade workflow locally with minimal coordination requirements, enabling fast parallel execution while reducing coordination complexity by eliminating the need for synchronized control across all machines.
Solution Approach 2:
Each machine performs self-upgrading through locally executed workflows that autonomously manage their own upgrade processes. The machines serve themselves without requiring external coordination or control, achieving high-speed parallel upgrades while minimizing coordination complexity through decentralized autonomous operation.
3Reliability
If multiple reboots are performed during upgrade, then system updates are completed, but the upgrade process becomes more time-consuming and complex to manage
Solution Approach 1:
The patent performs preliminary actions by saving the state of completed upgrades to state files before system reboots occur. This allows the upgrade process to resume after reboot without re-executing already-completed actions, reducing total upgrade time while ensuring reliable completion of all necessary updates across multiple reboot cycles.
Solution Approach 2:
The patent implements feedback mechanisms through state files that track which actions have been completed and which remain pending. This feedback enables the upgrade system to intelligently resume from the correct point after reboots, avoiding redundant operations and minimizing total upgrade time while ensuring all necessary updates are reliably completed.
4Reliability
If the upgrade process is interrupted or fails partway through, then manual intervention is required, but this increases operational complexity and reduces scalability
Solution Approach 1:
The patent segments the upgrade process into discrete, trackable actions with individual state records in state files. This segmentation enables automatic fault tolerance by allowing the system to identify and resume from the specific point of failure without requiring manual intervention, maintaining reliability while preserving operational simplicity through automated recovery.
Solution Approach 2:
The upgrade system performs self-recovery by automatically detecting failures and resuming from the point of interruption using state file information. This self-service capability eliminates the need for manual intervention during failures, enhancing fault tolerance while maintaining operational simplicity and enabling scalable deployment across large clusters.
Data Source
AI summary
A distributed reusable framework comprising a non-logging resumable workflow of actions that runs independently in parallel to upgrade each machine of a large distributed cluster of machines. A CHECK process and a RUN process are associated with each action in the workflow. The CHECK process determines dynamically the state of each action, and the RUN process brings an action to a completed state. Following a reboot on a machine, the workflow on the machine sequences back to its start and rechecks each prior action of the workflow to verify that each action is in a completed state before. Upon a failure of the upgrade on a single machine in the cluster, the failed machine can be fixed and the workflow can be resumed without redoing the work already done. Since the actual live state is verified on each machine of the cluster, log files are unnecessary.