Distributed Metadata Tiering for Scalable Storage Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face limitations in scalability and redundancy, particularly in centralized filesystems where metadata protection is inadequate, leading to data loss in case of server failure and inability to scale with increasing numbers of metadata servers.
Innovation Solution
A distributed storage system utilizing a virtual file system (VFS) that spreads metadata functionality across many servers, employing non-volatile memory (NVM) and tiered memory architecture to manage data distribution and redundancy, allowing for load balancing and efficient data management across thousands of nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized filesystem with metadata protection is used, then data security is improved, but scalability deteriorates
Solution Approach 1:
The patent segments the centralized metadata structure into distributed metadata servers across multiple nodes. Each node maintains a portion of the metadata, eliminating the single point of failure while enabling horizontal scaling. The filesystem is divided into multiple segments that can be independently managed and replicated.
Solution Approach 2:
The patent implements metadata replication across multiple servers to ensure data security and availability. Critical metadata is copied and distributed to redundant nodes, so that if one server fails, the metadata remains accessible from other copies, thereby improving reliability while enabling scalability.
2Adaptability or versatility
If number of metadata servers is increased, then scalability is improved, but system complexity worsens
Solution Approach 1:
The patent designs metadata servers with universal functionality that can perform multiple operations (read, write, replicate, recover) uniformly across all nodes. This standardized multi-functional approach allows the system to scale by simply adding more servers with the same capabilities, rather than managing increasingly complex specialized components.
Solution Approach 2:
The patent implements self-service mechanisms where metadata servers automatically replicate data, detect failures, and recover from errors without centralized coordination. Each server independently manages its own state and communicates with peers, reducing the complexity of centralized control while enabling scalable deployment.
3Adaptability or versatility
If distributed storage architecture is implemented, then scalability is improved, but data redundancy deteriorates
Solution Approach 1:
The patent implements explicit metadata copying and replication strategies in the distributed architecture. Critical metadata is replicated across multiple nodes with defined redundancy levels, ensuring that data redundancy is maintained even as the system scales distributed across numerous servers.
Solution Approach 2:
The patent employs prior cushioning by pre-replicating metadata to multiple locations before failures occur. This proactive redundancy ensures that when nodes are added or failures occur, data protection is already in place, maintaining reliability while supporting scalable distributed architecture.
Data Source
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. The storage devices may be assigned to one of a plurality of memory tiers, and the data in a storage device may be reassigned to another storage device in a different memory tier.