Multi-domain clustered file system architecture for failure isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file systems face challenges in minimizing the impact of system failures, particularly when multiple nodes or drives fail, leading to data unavailability and increased costs in identifying affected files, especially as the system size grows, and existing solutions either limit cluster size or create multiple clusters, which can complicate user management and fail to optimize deduplication performance across different storage tiers.
Innovation Solution
A multi-domain, multi-tier architecture is implemented, where the system is partitioned into sub-clusters (domains) with a file system redirection service managing a global namespace and redirecting file accesses to appropriate domains, allowing each domain to have different storage tiers with varying cost and performance characteristics, and supporting optimal deduplication performance while maintaining service level agreements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the cluster size is increased to provide more storage capacity and performance, then the system can handle larger workloads, but the probability of multiple node or drive failures increases, leading to data unavailability
Solution Approach 1:
The patent divides the distributed file system into multiple isolated domains, where each domain contains a subset of nodes and data. When failures occur, they are contained within specific domains rather than affecting the entire system. The file system redirection service routes client requests to appropriate domains, ensuring that failures in one domain do not propagate to others, thus maintaining data availability while supporting large cluster sizes.
2Reliability
If multiple clusters are created to limit failure impact, then reliability is improved, but device complexity and user management difficulty increase
Solution Approach 1:
The patent merges multiple isolated domains into a single unified namespace that clients interact with. The file system redirection service transparently manages domain routing, combining the benefits of multiple isolated clusters (failure containment) with the simplicity of a single cluster interface. Clients see and interact with one unified file system, while the redirection service handles the complexity of domain management internally.
3Quantity of substance
If the file system size grows to provide more storage, then capacity increases, but the cost and time of iterating the file system to identify affected files during failures increases
Solution Approach 1:
By organizing the file system into multiple domains with isolated namespaces, the patent limits the scope of file system iteration during failures to only the affected domain rather than the entire file system. When a failure occurs, the system only needs to identify and process files within the specific domain impacted, significantly reducing the time and computational cost compared to iterating through all files in a large unified file system.
4Quantity of substance
If deduplication is performed across the entire file system to optimize storage efficiency, then storage utilization improves, but the complexity of managing deduplication across multiple clusters increases
Solution Approach 1:
The patent enables deduplication to be performed across multiple domains by providing a unified namespace through the file system redirection service. This allows the system to achieve storage efficiency benefits of cross-domain deduplication while the redirection service transparently manages the complexity of coordinating deduplication operations across domain boundaries, presenting a simplified interface to clients and administrators.
Data Source
AI summary
Embodiments are described for a multi-domain and multi-tier architecture for clustered network file systems. This system allows a user to create sub-cluster of physical nodes, called domains, and file system resources for the data placed in a domain are allocated only from the nodes in the domain. It limits the impact of system failures to the files within a domain. A file system redirection service manages a global namespace spanning the domains and redirects file accesses to the appropriate domain where it is stored. In each domain, there are different classes of storage, tiers, with different cost and performance characteristics. Files can be placed on a set of tiers depending on a storage level agreement (SLA) specified for a file. Tier examples include a higher performance tier consisting of SSDs and a lower performance tier of HDDs.