Coalescing Metadata Chains for Cluster Storage Mirroring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cluster storage systems, the mirroring of metadata sets from a local node to a remote node results in significant network congestion and bandwidth consumption, as each metadata set is typically sent separately, leading to inefficiencies in data transmission and storage.
Innovation Solution
The system coalesces metadata sets into metadata chains, where only a representative metadata set is sent to the remote node, reducing the number of metadata sets transmitted and stored, thereby minimizing network congestion and bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each metadata set is sent separately to the remote node, then data integrity and availability are maintained, but network congestion and bandwidth consumption increase significantly
Solution Approach 1:
The patent groups multiple metadata sets into a single metadata chain structure, where related metadata sets are combined and transmitted together rather than individually. This merging approach reduces the total number of transmission operations and conserves network bandwidth while preserving all necessary data through the chain structure.
Solution Approach 2:
The system performs preliminary grouping and organization of metadata sets into chains before transmission to the remote node. By pre-organizing metadata sets that belong together into chain structures locally, the system optimizes transmission efficiency and reduces network congestion before the actual data transfer occurs.
2Loss of information
If each metadata set is sent separately to the remote node, then complete metadata information is preserved, but the number of transmission operations increases
Solution Approach 1:
Multiple metadata sets are merged into a single metadata chain transmission unit, reducing the number of separate transmission operations. The chain structure internally maintains references to all individual metadata sets, ensuring complete information preservation while improving transmission productivity through batched operations.
Solution Approach 2:
The patent changes the transmission parameter from individual metadata set units to aggregated metadata chain units. This parameter change in how metadata is packaged and transmitted reduces the frequency of transmission operations while maintaining information completeness through the chain's internal structure.
3Reliability
If all metadata sets are transmitted to the remote node, then data availability is ensured, but network congestion increases
Solution Approach 1:
The patent combines multiple metadata sets into metadata chains, transmitting them as consolidated units rather than separate entities. This merging reduces the total number of transmission events and associated network overhead, thereby reducing congestion while maintaining data availability through the preserved chain structure.
Solution Approach 2:
The metadata chain structure acts as an intermediary mechanism between the local and remote nodes. It consolidates multiple metadata sets into a single transmission vehicle, reducing direct transmission overhead and network congestion while ensuring all underlying metadata remains accessible through the chain's internal references.
Data Source
AI summary
Described herein are a system and method for remote mirroring/copying data and metadata sets from a local node to a remote node that reduces the number of metadata sets that are mirrored. In some embodiments, the local node may coalesce metadata sets into metadata chains, each metadata chain comprising a grouping of two or more metadata sets. In some instances, a “representative” metadata set of a metadata chain may be selected for sending to the remote node for storing, wherein the other metadata sets of the metadata chain are not sent to the remote node. In these embodiments, the selected metadata set may represent all the metadata sets in the chain and be the only metadata set in the chain that is transmitted and stored to the remote node. As such, the network congestion between the local and remote nodes may be reduced.