Cluster Drive Firmware Updates With Failure-Domain Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cluster-wide drive firmware update methods in storage clusters are inefficient, requiring serial node-by-node and drive-by-drive updates, leading to prolonged completion times and potential data unavailability due to serial rebalancing of mirrored partitions.
Innovation Solution
Implementing a reservation-based neighborhood-wide lock system that allows parallel updates of multiple drives within a neighborhood, ensuring data availability and minimizing rebalancing operations to reduce completion time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If serial node-by-node and drive-by-drive firmware updates are performed, then data availability is maintained, but update completion time is prolonged
Solution Approach 1:
The patent segments the cluster into failure domains and further into neighborhoods (groups of drives). By updating drives within the same neighborhood in parallel while maintaining serial updates across different failure domains, the system achieves both speedup and data availability. The segmentation allows independent parallel execution within safe boundaries.
Solution Approach 2:
The patent introduces a hierarchical dimension to the update process: failure domains at the top level (updated serially for safety) and neighborhoods within domains at a lower level (updated in parallel for speed). This dimensional hierarchy resolves the contradiction by allowing parallelism where safe and serial execution where critical.
2Loss of time
If parallel firmware updates are performed across multiple drives, then update completion time is reduced, but system complexity increases due to coordination requirements
Solution Approach 1:
By segmenting drives into neighborhoods within failure domains, the patent reduces coordination complexity. Each neighborhood can be updated independently in parallel, and the segmentation provides natural boundaries that simplify lock management and coordination compared to a fully parallel approach across all drives.
Solution Approach 2:
The patent uses reservations as an intermediary mechanism to coordinate parallel updates. A node obtains a reservation for a failure domain before updating drives within it, acting as a mediator that simplifies coordination between multiple nodes and drives without requiring complex inter-node communication protocols.
3Reliability
If serial rebalancing of mirrored partitions is performed during firmware updates, then data integrity is preserved, but rebalancing time increases completion time
Solution Approach 1:
The patent merges multiple rebalancing operations into a single rebalance performed after all parallel drive updates within a neighborhood are complete. Instead of rebalancing after each individual drive update, the combined approach maintains data integrity while reducing the total time spent on rebalancing operations.
Solution Approach 2:
The patent maintains continuous useful action by performing drive firmware updates in parallel without interrupting other operations. The rebalancing is performed continuously after updates complete, rather than sequentially between each update, maximizing system utilization and reducing total completion time.
Data Source
AI summary
A system can maintain a computer cluster that comprises a group of nodes, wherein a node of the group of nodes comprises a group of storage drives, wherein the node is a member of a failure domain that comprises a subgroup of nodes of the group of nodes, wherein the failure domain is configured to preserve data stored in the failure domain when at least one node within the failure domain fails. The system can obtain a reservation for the node, wherein the reservation permits the node to make the group of storage drives unavailable for data access, and wherein other nodes within the failure domain are unable to obtain the reservation while the node possesses the reservation. The system can, while the node possesses the reservation, update firmware for respective storage drives for the group of storage drives in parallel.