Distributed File System Disaggregation for Scalable Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed file systems face challenges in scaling reliably while maintaining high resiliency, load balancing, and managing workloads efficiently, often resulting in increased costs and longer latency, and are unable to protect against node or drive failures.
Innovation Solution
A distributed file system with disaggregated data and storage management subsystems, allowing independent scaling and decoupling of data management and storage management layers, enabling on-demand resource deployment, load balancing, and self-healing capabilities, with a software-defined architecture that supports hardware agnosticism and multi-protocol access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed file systems use traditional monolithic architecture, then system simplicity is maintained, but scaling reliability and resiliency are compromised
Solution Approach 1:
The system divides the distributed file system into separate data management and storage management layers. The data management layer handles file system operations, metadata, and client requests, while the storage management layer handles actual data storage, replication, and recovery. This segmentation allows independent scaling and optimization of each layer, improving reliability without overwhelming complexity.
Solution Approach 2:
The patent extracts the storage management functions from the traditional monolithic file system architecture, creating a separate storage layer that can be independently managed, scaled, and optimized. This extraction enables the data management layer to focus on file system operations while storage management handles durability and replication, improving overall system reliability.
2Quantity of substance
If distributed file systems scale to handle more clients and objects, then system capacity increases, but load balancing and resource management become more difficult
Solution Approach 1:
The patent introduces a distributed metadata service as an intermediary between clients and storage nodes. This metadata service maintains information about data location, replication status, and node capacity, enabling automated load balancing and resource allocation. The metadata layer abstracts the complexity of distributed resource management, making it easier to scale the system while maintaining balanced resource utilization.
3Quantity of substance
If distributed file systems use existing scaling techniques, then system capacity increases, but write and read latencies increase
Solution Approach 1:
The system performs preliminary actions by pre-caching frequently accessed metadata in memory and pre-positioning data replicas across multiple nodes before they are needed. The metadata service maintains up-to-date information about data locations and node status, so when read or write operations occur, the system can immediately route requests to the appropriate nodes without scanning or searching, thereby reducing latency while maintaining high capacity.
4Quantity of substance
If distributed file systems prioritize scaling, then system capacity increases, but protection against node and drive failures is compromised
Solution Approach 1:
The patent implements beforehand cushioning by maintaining multiple replicas of each data block across different nodes and drives. When data is written, the system immediately creates and distributes replicas to designated storage nodes, ensuring that even if the primary node or drive fails, the data remains accessible from replicas. This redundancy is established in advance, providing automatic failure protection that scales with system capacity.
5Productivity
If distributed file systems use integrated data and storage management, then system simplicity is maintained, but resource utilization and efficiency are optimized poorly
Solution Approach 1:
The patent implements dynamic resource allocation by decoupling data management from storage management. The storage management layer dynamically monitors node capacity, performance, and health status, automatically allocating storage resources and adjusting replication factors based on current system conditions. This dynamic approach enables optimal resource utilization while the separate management layers keep complexity manageable through specialized functions.
Data Source
AI summary
Systems and methods for managing data storage using a distributed file system are provided. In one example, a file system instance is deployed virtually in a node of a distributed storage system. The file system instance has a dynamic configuration including a set of services corresponding to a cluster management subsystem and a storage management subsystem. The storage management subsystem operates independently of a data management subsystem of the distributed storage system as a result of disaggregation from the data management subsystem. The data management subsystem performs storage and block management functions based on requests received from an application layer. An additional service corresponding to either the data management subsystem or the storage management subsystem is deployed virtually to meet the demand for the additional service in response to determining the presence of a demand for the additional service and availability a set of resources corresponding to the additional service.