Incremental Metadata Refresh for Distributed Database Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face challenges in maintaining metadata integrity and availability during node failures, leading to prolonged recovery times and performance costs in large clusters.
Innovation Solution
An incremental refresh technique is implemented in a redundant array of independent nodes, where metadata objects are stored in regions with various states, allowing for the creation of a partial region copy, updating it with pending updates, and converting it to a backup region to quickly restore and synchronize data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional full synchronization methods are used to restore metadata regions after node failure, then data integrity is maintained, but cluster recovery time increases and resource consumption rises
Solution Approach 1:
The patent segments the metadata region restoration process into two distinct phases: (1) rapid initialization by copying only the region structure and metadata from the authoritative region to create a functional backup region immediately, and (2) incremental data synchronization that occurs in the background. This segmentation allows the system to achieve fast recovery while maintaining data integrity through progressive synchronization.
Solution Approach 2:
The patent applies preliminary action by pre-establishing the backup region structure and copying essential metadata before the node fully returns to the cluster. This allows the backup region to be operational immediately upon node recovery, reducing overall recovery time while data synchronization continues in the background to ensure complete data integrity.
2Reliability
If full data synchronization is performed when a node rejoins the cluster, then data consistency is ensured, but system performance and resource utilization deteriorate
Solution Approach 1:
The patent implements periodic action through incremental data synchronization that occurs in scheduled intervals after the backup region is initially created. Instead of performing a single large-scale data copy that would overwhelm system resources, the synchronization happens periodically in manageable chunks, maintaining data consistency while preserving system performance during the node reintegration process.
3Reliability
If complete region copying is performed to create backup regions, then data redundancy is achieved, but resource consumption and operational complexity increase
Solution Approach 1:
The patent extracts only the essential elements needed for immediate backup region functionality - specifically the region structure and metadata - and separates them from the complete data set. This allows the backup region to become operational quickly with minimal resource consumption, while the full data redundancy is achieved progressively through incremental synchronization, thereby reducing operational complexity.
Data Source
Figure 1
Figure 2~4
Figure 3
AI summary
Cluster recovery time is reduced by an incremental refresh technique that is described herein. The goal of the technique is to salvage the portion of the redundant distributed database lost (e.g., during a fault) by performing an incremental refresh of only the updates occurring to that portion of the database since the time of loss.