Coalescing Metadata for Remote Storage Mirroring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cluster storage systems, the mirroring of metadata between local and remote nodes leads to significant network congestion and bandwidth consumption due to the need to transmit multiple metadata sets, which is inefficient and can cause data availability issues if not managed effectively.
Innovation Solution
The system coalesces metadata sets into metadata chains, where only a representative metadata set is sent to the remote node for storage, reducing the number of metadata sets transmitted and stored, thereby minimizing network congestion and bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple metadata sets are transmitted to the remote node for mirroring, then data availability and consistency are improved, but network congestion and bandwidth consumption increase significantly
Solution Approach 1:
The patent merges multiple metadata sets into a single consolidated metadata set that contains references to multiple data sets. Instead of transmitting separate metadata for each data set, the system transmits one unified metadata structure that points to all relevant data sets, thereby reducing network bandwidth consumption while maintaining data availability and consistency at the remote node
2Reliability
If multiple metadata sets are transmitted to the remote node, then data consistency is maintained, but network congestion increases
Solution Approach 1:
The patent combines multiple metadata sets into a single consolidated metadata structure that references multiple data sets. This merging approach reduces the total number of metadata transmissions over the network, thereby reducing network congestion while preserving data consistency through the unified metadata references
3Reliability
If all metadata sets are mirrored to the remote node, then data availability is ensured, but storage space and processing resources are consumed excessively
Solution Approach 1:
The patent extracts only the essential metadata information needed for data availability and consistency, separating it from redundant metadata. By taking out only the critical metadata references and transmitting just those to the remote node, the system ensures data availability while minimizing storage space and processing resource consumption
Data Source
AI summary
Described herein are a system and method for remote mirroring/copying data and metadata sets from a local node to a remote node that reduces the number of metadata sets that are mirrored. In some embodiments, the local node may coalesce metadata sets into metadata chains, each metadata chain comprising a grouping of two or more metadata sets. In some instances, a “representative” metadata set of a metadata chain may be selected for sending to the remote node for storing, wherein the other metadata sets of the metadata chain are not sent to the remote node. In these embodiments, the selected metadata set may represent all the metadata sets in the chain and be the only metadata set in the chain that is transmitted and stored to the remote node. As such, the network congestion between the local and remote nodes may be reduced.