Distributed Filesystem Across Failure Domains for Resilient Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face limitations in scalability and resilience, particularly when a single server failure leads to data loss due to lack of adequate protection mechanisms, and existing clustered systems are unable to efficiently distribute metadata across multiple nodes.
Innovation Solution
A distributed filesystem architecture that spans multiple failure domains, utilizing non-volatile memory (NVM) and distributed erasure coding, allows metadata to be spread across many servers, enabling load distribution and independent operations without coordination, and employs forward error correction to ensure data resilience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in a centralized filesystem on a single server, then the system is simple to manage, but the system loses data availability when the server fails
Solution Approach 1:
The patent segments the centralized filesystem into a distributed filesystem where metadata is divided and stored across multiple independent nodes. Each node maintains a portion of the metadata, eliminating the single point of failure while distributing the storage burden across the network.
Solution Approach 2:
The patent implements copying by replicating metadata across multiple nodes in the distributed filesystem. Each node holds copies or portions of the metadata, ensuring that data remains accessible even if individual nodes fail, thereby improving reliability through redundancy.
2Productivity
If metadata is stored on a single server, then the storage system is simple, but the system cannot scale efficiently and loses data when the server fails
Solution Approach 1:
The patent segments metadata into discrete units that can be independently distributed across multiple nodes. This segmentation enables the system to scale horizontally by adding more nodes, as each node can independently store and manage portions of the metadata without requiring centralized coordination.
Solution Approach 2:
The patent transitions from a single-dimension centralized metadata storage model to a multi-dimensional distributed model across multiple nodes. This dimensional change allows the system to scale by adding nodes in different dimensions of the network, improving productivity through enhanced scalability.
3Productivity
If a single server is used for filesystem operations, then the system is easy to operate, but the system experiences performance bottlenecks and data loss on failure
Solution Approach 1:
The patent implements self-service by enabling each node in the distributed filesystem to independently perform metadata operations without requiring constant coordination with other nodes. Each node autonomously manages its stored metadata, eliminating performance bottlenecks while maintaining operational simplicity through decentralized autonomy.
4Reliability
If data is replicated across multiple servers, then data availability improves, but the complexity of managing distributed data increases
Solution Approach 1:
The patent segments data and metadata into manageable units distributed across nodes, where each node independently manages its portion. This segmentation reduces the complexity of distributed management by localizing control at each node while maintaining overall system reliability through the segmented distribution architecture.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A plurality of failure domains are communicatively coupled to each other via a network, and each of the plurality of failure domains is coupled to one or more storage devices. A failure resilient stripe is distributed across the plurality of storage devices, such that two or more blocks of the failure resilient stripe are located in each failure domain.