Point-in-Time Snapshots in Backend Object Storage with Sharded Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file systems face scalability limitations and data loss risks due to centralized metadata management, especially in large clusters, and lack efficient methods for distributing load and protecting data across multiple nodes.
Innovation Solution
A distributed virtual file system architecture that utilizes non-volatile memory (NVM) and metadata shards spread across many servers, enabling load distribution and snapshot management to backend object storage for redundancy and scalability, with each bucket managing its own erasure coding and snapshot operations independently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized metadata management is used in conventional file systems, then data integrity and access control are maintained, but scalability is limited and single point of failure risk increases
Solution Approach 1:
The patent divides the centralized metadata management into distributed metadata shards stored across multiple nodes. Each metadata shard is independently managed by different nodes, eliminating the single point of failure while maintaining data integrity through distributed consensus mechanisms. This segmentation enables the system to scale horizontally by adding more nodes without compromising metadata management capabilities.
2Ease of operation
If centralized metadata management is used, then data access control is simplified, but load distribution becomes inefficient in large clusters
Solution Approach 1:
The metadata management function is segmented into multiple independent shards distributed across different nodes. Each shard handles a specific portion of the data, enabling parallel processing and efficient load distribution. The system maintains access control through distributed authentication mechanisms that operate across all shards without requiring centralized authorization.
3Device complexity
If metadata is stored on fewer nodes, then system complexity is reduced, but availability and fault tolerance decrease
Solution Approach 1:
The patent implements local quality by having each metadata shard stored on specific nodes with replicated copies on other nodes. Each node manages its local shard independently with appropriate access controls and replication policies. This approach maintains manageable complexity at each node while ensuring high availability through replication, as data remains accessible even if individual nodes fail.
Data Source
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. The computing devices may push a point in time to a backend for a distributed storage system.