Erasure-Coded Data Recovery with Concurrent Multi-Node Repair
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face performance bottlenecks in data recovery due to resource-intensive metadata queries and updates when handling failures, especially with large numbers of data objects and nodes, leading to inefficient disk operations and network bandwidth usage.
Innovation Solution
The Virtual Chunk Service (VCS) based storage technique splits storage nodes into smaller failure domains (VCSs) with unique IDs, allowing concurrent restoration of data fragments across multiple failed nodes within an Erasure Coding Group (ECG), reducing the need for repeated reads and metadata updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the distributed data storage system queries metadata for each data object to identify impacted objects during node failure, then data recovery can be initiated, but the process becomes resource intensive and creates performance bottlenecks
Solution Approach 1:
The patent segments the metadata system into a metadata service separate from data nodes. The metadata service maintains a mapping between data objects and storage nodes, allowing the system to identify impacted objects without querying each object's metadata individually. This segmentation enables efficient identification of affected data objects during node failure while maintaining data recovery capability.
2Loss of information
If the system reads metadata of all data objects to identify those stored on the failed node, then complete identification is achieved, but the process becomes inefficient with billions of data objects
Solution Approach 1:
The patent introduces a metadata service as an intermediary between data nodes and the recovery process. This metadata service maintains a global mapping of data objects to storage nodes, allowing the system to efficiently identify which data objects are stored on failed nodes without reading through all object metadata. The intermediary provides O(1) lookup capability for identifying impacted objects.
3Reliability
If metadata updates are performed for each impacted object after repair, then data integrity is maintained, but the process consumes significant computing resources
Solution Approach 1:
The patent merges the metadata update operation into a single centralized update at the metadata service level, rather than updating metadata for each individual data object. When a node fails or is repaired, the metadata service updates its internal mapping once to reflect the new storage location, eliminating the need for repeated metadata updates across billions of objects while maintaining data integrity.
4Quantity of substance
If the distributed data storage system handles a significantly large number of data objects, then storage capacity is improved, but metadata queries and updates cause serious performance issues
Solution Approach 1:
The patent introduces a new dimensional layer - the metadata service layer - that operates independently from the data object layer. This metadata service maintains a projected view of data object locations without requiring access to each individual data object. This dimensional separation allows the system to handle billions of data objects efficiently by operating on the metadata layer rather than the data object layer during recovery operations.
Data Source
AI summary
To efficiently recover from a multiple storage node failure, a storage node concurrently restores data fragments to the multiple failed storage nodes, as opposed to restoring each node individually. In the VCS based storage technique, storage nodes are restored as part of an ECG repair process. For each ECG being repaired, a storage node performing the restoration process reads data fragments from active nodes in the ECG and generates new data fragments to replace any lost data fragments. The node then stores one of the new data fragments across each of the failed storage nodes. By concurrently restoring data fragments to each failed storage node, the data fragments needed to repair each ECG are only read once, thereby preserving disk operations and network bandwidth.