Distributed File Page Caching for Coherent Metadata Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional filesystems face scalability limitations in metadata management, especially in clustered systems, leading to potential data loss and inefficiencies in data access and storage, particularly in large-scale distributed environments.
Innovation Solution
A distributed virtual file system architecture that utilizes non-volatile memory (NVM) and volatile memory (RAM) to manage metadata across multiple servers, employing distributed failure resilient address spaces (DFRAS) and lease-based caching mechanisms to optimize data access and storage, allowing for load distribution and high-performance parallel commits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If centralized metadata management is used in conventional filesystems, then metadata access is simple and consistent, but scalability is limited and system reliability decreases in clustered environments
Solution Approach 1:
The patent segments metadata management by distributing it across multiple nodes in a clustered filesystem. Each node maintains its own metadata cache and serves requests locally, while coherence protocols ensure consistency across the distributed metadata structure. This eliminates the single point of failure in centralized metadata management while maintaining operational simplicity through automated coherence maintenance.
2Reliability
If centralized metadata management is used, then consistency is easier to maintain, but system scalability is limited in large-scale distributed environments
Solution Approach 1:
The system segments metadata consistency into distributed coherence protocols that operate locally on each node while maintaining global consistency. This allows linear scalability with the number of nodes, as each node independently manages its metadata cache without requiring centralized coordination for every operation.
Solution Approach 2:
The patent introduces coherence protocols as intermediaries between distributed metadata caches. These protocols automatically maintain consistency across nodes without requiring explicit consistency management from applications, enabling scalability while preserving consistency guarantees.
3Speed
If page cache is implemented in distributed filesystems, then data access performance improves, but coherency maintenance becomes more difficult
Solution Approach 1:
The coherence protocol operates autonomously on each node, automatically detecting and resolving coherence conflicts without external intervention. When a node modifies its local page cache, it autonomously invalidates corresponding caches on other nodes through distributed notifications, eliminating the need for centralized coherency management.
Solution Approach 2:
The system implements feedback mechanisms where nodes continuously monitor cache validity and propagate coherence status changes across the distributed system. This feedback loop enables automatic coherency maintenance while preserving high-speed local cache access, as modifications are immediately propagated to all relevant nodes.
4Productivity
If distributed metadata management is implemented, then system scalability improves, but data loss risk increases without proper protection mechanisms
Solution Approach 1:
The patent implements beforehand cushioning by maintaining redundant metadata copies across multiple nodes before failures occur. Each node caches metadata locally and automatically replicates it to other nodes, creating a cushion against data loss. This redundancy is established proactively through distributed caching operations, ensuring protection against node failures without requiring additional active management.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. The computing devices may use local caches in a coherent manner when accessing the plurality of storage devices.