ECG Manager Election for Erasure-Coded Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage systems face performance bottlenecks in data recovery due to resource-intensive metadata queries and updates when a storage node fails, especially with large numbers of data objects and nodes, leading to inefficiencies in identifying and repairing impacted data fragments.
Innovation Solution
The implementation of a virtual chunk service (VCS) based storage technique, which splits storage nodes into multiple VCSs with unique IDs, forms data storage groups, and uses erasure coding to manage data fragments across nodes, minimizing metadata updates and queries by reassigning VCSs to alternate nodes during recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional metadata-based data recovery is used, then data reliability is maintained, but system performance and recovery efficiency deteriorate due to resource-intensive metadata queries and updates
Solution Approach 1:
The patent segments the distributed storage system into multiple erasure coding groups (ECGs), each managed by a dedicated manager node. This segmentation allows the system to track data fragments at the ECG level rather than requiring metadata queries for each individual data object, significantly reducing the overhead during recovery operations while maintaining data reliability across the distributed system.
2Reliability
If metadata queries are performed for each data object to identify impacted objects, then complete data recovery is ensured, but computing resources and time are excessively consumed
Solution Approach 1:
The patent introduces erasure coding group managers as intermediary nodes that maintain metadata about data fragments stored in each ECG. Instead of querying metadata for every data object during recovery, the system queries the ECG manager, which provides aggregated information about all fragments in the group. This intermediary approach ensures complete recovery while reducing computing resource consumption by eliminating redundant metadata queries.
3Reliability
If metadata updates are performed for each impacted object after repair, then data integrity is maintained, but system performance deteriorates due to the large number of update operations
Solution Approach 1:
The patent merges the metadata update operations by having the erasure coding group manager maintain a consolidated view of all data fragments within an ECG. When repair operations are performed, the ECG manager updates the group-level metadata once, rather than requiring individual updates for each data object. This merging of update operations maintains data integrity while significantly improving system performance during recovery processes.
Data Source
AI summary
To ensure that there is an elected manager among storage nodes of an erasure coding group (“ECG”), an ECG manager (“ECGM”) election process is periodically performed among available storage nodes that are configured with the software to perform the services of an ECGM. When a storage node is activated, an ECGM process of the storage node begins executing and is assigned a process identifier (“PID”). A storage node can utilize a service query framework to identify other available storage nodes and retrieve their ECGM PIDs. The storage node then selects a PID according to a criterion and elects the storage node corresponding to the selected PID to be the acting ECGM. This process is performed periodically, so even if the acting ECGM storage node fails, a new ECGM is eventually selected from the available storage nodes.