Distributed File Storage Service with Modular Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed file systems face challenges in maintaining high availability and durability while supporting large-scale, concurrent access to files across thousands of clients, particularly due to limitations in industry-standard file system interfaces that struggle with asynchronous interactions, component failures, and network delays.
Innovation Solution
A modular architecture for a distributed file storage service that separates metadata and data into distinct subsystems, using replication techniques like erasure coding and consensus-based state machines to ensure sequential consistency and durability, while allowing for independent scaling and failover of access, metadata, and storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If industry-standard file system interfaces are used, then compatibility and ease of operation are improved, but reliability and consistency deteriorate due to asynchronous interactions, component failures, and network delays
Solution Approach 1:
The patent introduces a distributed file system interface layer that acts as an intermediary between standard file system operations and the underlying distributed storage system. This interface translates standard file operations into distributed operations that maintain consistency models, handle failures, and manage asynchronous interactions across network partitions, thereby preserving both ease of operation and reliability
2Productivity
If storage capacity is increased to support more clients, then productivity is improved, but device complexity increases
Solution Approach 1:
The patent segments the distributed file storage system into independent modular components including metadata servers, data servers, and clients. Each component can be independently scaled and managed, allowing the system to support more clients and increase productivity while avoiding exponential complexity growth through structured decomposition
3Reliability
If data is replicated across multiple nodes for durability, then reliability is improved, but loss of time increases due to synchronization overhead
Solution Approach 1:
The patent implements periodic replication where data is replicated to multiple nodes asynchronously rather than synchronously for every operation. This periodic replication strategy maintains durability by ensuring eventual consistency while reducing the time penalty associated with real-time synchronization across all replicated nodes
Data Source
AI summary
In response to receiving a write request directed to a particular logical block of a storage object, a page of free space (sufficient to accommodate the payload of the write request, but smaller in size than the logical block) of a particular extent that has been selected to store contents of the logical block is allocated. The current size of the extent is smaller than the combined sizes of logical blocks that are mapped to the extent. The page is modified in accordance with a payload indicated in the write request. In response to a subsequent write request directed to the particular extent, a determination is made that the particular extent would violate a free space threshold criterion if the payload of the write request were accommodated, and an extent expansion operation is initiated.