Erasure-Coded Fragment Repair by Storage Node Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face challenges in reliable recovery of erasure coded data due to the unreliability of centralized metadata repositories, which can lead to metadata loss and query failures during storage node failures, making it difficult to determine missing fragments for recovery.
Innovation Solution
A distributed storage system that generates a list of missing fragments by scanning storage devices of available nodes rather than relying on a centralized metadata database, using a storage node to identify affected erasure coding groups, determine available nodes, and merge fragment lists from multiple nodes to create a master list for recovery, thereby ensuring reliable recovery without relying on a centralized metadata system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a centralized metadata repository is used to track erasure coded fragments, then data management and recovery coordination are simplified, but the system becomes vulnerable to metadata loss and query failures during storage node failures
Solution Approach 1:
The patent extracts the metadata dependency from the recovery process by having nodes scan their own storage devices directly to identify missing fragments, rather than querying a centralized metadata repository. This isolation removes the single point of failure while maintaining recovery functionality.
Solution Approach 2:
Storage nodes perform self-service by autonomously scanning their own storage devices to identify which erasure coded fragments they hold and which are missing, eliminating the need to query external metadata repositories during failure scenarios.
2Productivity
If centralized metadata queries are used to identify missing fragments, then the recovery process is streamlined, but query failures occur when metadata repositories are unavailable
Solution Approach 1:
Storage nodes continuously maintain up-to-date information about their stored fragments through preliminary scanning operations, so when recovery is needed, they can immediately provide accurate fragment location information without requiring metadata queries during the failure event.
Solution Approach 2:
The patent introduces peer-to-peer communication between storage nodes as an intermediary mechanism, where available nodes can share fragment location information directly, bypassing the need for centralized metadata queries while maintaining recovery efficiency.
3Measurement precision
If fragment lists are collected from multiple storage nodes and merged, then comprehensive recovery information is obtained, but the complexity of coordinating multiple nodes increases
Solution Approach 1:
The patent segments the fragment identification task by having each storage node independently scan and report its own fragment inventory, then merging these segmented results at a coordination point. This division of labor maintains accuracy while managing complexity through modular information collection.
Solution Approach 2:
Storage nodes merge their individual fragment lists into a comprehensive view of available fragments across the distributed system, enabling accurate identification of missing fragments while distributing the computational workload across multiple nodes rather than centralizing it.
Data Source
AI summary
Distributed storage systems frequently use a centralized metadata repository that stores metadata in an eventually consistent distributed database. However, a metadata repository cannot be relied upon for determining which erasure coded fragments are lost because of a storage node(s) failures. Instead, when recovering a failed storage node, a list of missing fragments is generated based on fragments stored in storage devices of available storage nodes. A storage node performing the recovery sends a request to one or more of the available storage nodes for a fragment list. The fragment list is generated, not based on a metadata database, but on scanning storage devices for fragments related to the failed storage node. The storage node performing the recovery merges retrieved lists to create a master list indicating fragments that should be regenerated for recovery of the failed storage node(s).