Erasure-Coded Storage Recovery with Concurrent Fragment Rebuild
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face performance bottlenecks during data recovery due to resource-intensive metadata queries and updates when handling node or volume failures, especially in systems with billions of data objects, leading to inefficiencies in identifying and repairing impacted data fragments.
Innovation Solution
The implementation of a virtual chunk service (VCS) based storage technique, which splits storage nodes into smaller failure domains (VCSs) and manages them as part of erasure coding groups, allowing concurrent restoration of data fragments across multiple failed storage nodes, reducing the need for repeated reads and metadata updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional metadata-based recovery approach is used to identify and repair data objects after node failure, then data reliability is maintained, but system performance deteriorates due to resource-intensive metadata queries and updates across billions of data objects
Solution Approach 1:
The patent segments the system into Virtual Chunk Services (VCS) that independently manage specific data fragments. Instead of querying metadata for all data objects, the system only needs to query the VCS managing the specific failed fragment, reducing the recovery scope from billions of objects to a small subset managed by a single VCS.
Solution Approach 2:
The VCS acts as an intermediary layer between the storage nodes and the recovery process. Each VCS maintains local metadata about its managed fragments, serving as a mediator that eliminates the need for global metadata queries. The VCS coordinates fragment recovery by selecting replacement nodes and orchestrating the data transfer.
2Reliability
If global metadata is queried for each data object to identify impacted objects during node failure, then complete data recovery is achieved, but computing resources are excessively consumed
Solution Approach 1:
The metadata management is segmented and distributed across multiple VCS instances, each maintaining metadata only for its specific data fragments. This eliminates the need to query a single global metadata store for all data objects, reducing computing resource consumption from O(n) to O(1) where n is the total number of data objects.
Solution Approach 2:
Each VCS has local knowledge about its managed fragments through local metadata, rather than relying on global metadata. This local quality enables the system to perform recovery operations using only locally available information, significantly reducing computing resource requirements.
3Reliability
If metadata updates are performed for each impacted data object after fragment recovery, then data integrity is maintained, but network bandwidth and time are wasted due to repeated updates
Solution Approach 1:
The patent merges the metadata update operations into a single consolidated update performed by the VCS. Instead of updating metadata for each individual data object separately, the VCS performs a unified metadata update that reflects the new fragment location, eliminating redundant network operations and reducing recovery time.
Solution Approach 2:
The VCS autonomously manages metadata updates without requiring external coordination for each data object. The VCS automatically updates its local metadata and notifies relevant parties, enabling self-service that eliminates time-consuming coordinated update operations across the distributed system.
Data Source
AI summary
To efficiently recover from a multiple storage node failure, a storage node concurrently restores data fragments to the multiple failed storage nodes, as opposed to restoring each node individually. In the VCS based storage technique, storage nodes are restored as part of an ECG repair process. For each ECG being repaired, a storage node performing the restoration process reads data fragments from active nodes in the ECG and generates new data fragments to replace any lost data fragments. The node then stores one of the new data fragments across each of the failed storage nodes. By concurrently restoring data fragments to each failed storage node, the data fragments needed to repair each ECG are only read once, thereby preserving disk operations and network bandwidth.