Distributed Snapshot Storage with Scalable Metadata Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file systems face scalability issues due to limited metadata servers, leading to potential data loss in case of server failure and inefficiencies in load distribution, especially in large clusters.
Innovation Solution
A distributed virtual file system architecture that utilizes non-volatile memory and metadata distribution across multiple servers, allowing for scalable and resilient storage with independent bucket ownership and distributed erasure coding, enabling high-performance parallel commits and snapshot management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a centralized metadata server architecture is used, then the file system is simple to manage, but the system cannot scale to large clusters and is vulnerable to single points of failure
Solution Approach 1:
The patent divides the centralized metadata server into multiple distributed metadata servers, each responsible for specific metadata operations. This segmentation eliminates the single point of failure while maintaining manageable complexity through modular architecture where each metadata server operates independently but cooperatively within the distributed file system.
2Reliability
If multiple metadata servers are used in a clustered system, then reliability improves, but the system cannot scale to large clusters due to limited metadata servers
Solution Approach 1:
The patent introduces a new dimension to metadata server organization by implementing a hierarchical structure with primary and secondary metadata servers. This dimensional change allows the system to scale horizontally across large clusters while maintaining reliability through the multi-level redundancy architecture, enabling thousands of nodes to be managed efficiently.
3Reliability
If data is replicated across multiple storage nodes, then data availability improves, but storage efficiency decreases due to redundant data copies
Solution Approach 1:
The patent transforms the replication parameter from full data copying to erasure coding parameters, where data is divided into fragments with redundant parity information. This parameter change allows the system to achieve the same data availability as replication while using significantly less storage capacity, as only a portion of the stored data is actual user data while the rest is distributed redundancy information.
Data Source
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. The computing devices may take snapshots to store points in time coherently for a distributed storage system.