Distributed Key-Value Metadata File System
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed file-systems face scalability, performance, and fault tolerance limitations due to reliance on a single-master design for metadata management, which becomes inefficient with large datasets and real-time, low-latency requirements.
Innovation Solution
A distributed file-system architecture that uses a distributed key-value store to store metadata, employing unique inode numbers and composite row keys for files and directories, allowing for atomic operations and scalable storage across multiple servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single-master design is used for metadata management, then file-system operations can be processed centrally, but scalability and performance deteriorate as the system grows to handle large datasets
Solution Approach 1:
The patent segments the centralized metadata storage into distributed key-value stores across multiple servers. Each server maintains a portion of the metadata namespace, eliminating the single-master bottleneck. The segmentation is achieved through namespace partitioning where different servers handle different ranges or types of metadata keys, allowing parallel processing of file-system operations across the distributed cluster.
2Adaptability or versatility
If metadata is distributed across multiple servers, then scalability improves, but maintaining atomicity and consistency becomes more difficult
Solution Approach 1:
The patent introduces an intermediary layer of distributed consensus protocols and coordination mechanisms between the distributed key-value stores. This intermediary ensures that operations spanning multiple servers maintain atomicity through coordinated transactions, while consistency is enforced through versioning schemes and conflict resolution protocols. The intermediary abstracts the complexity of distributed coordination from the file-system operations.
Solution Approach 2:
The patent changes the operational parameters of the distributed key-value stores to optimize for both scalability and consistency. This includes adjusting replication factors, consistency levels, and transaction timeout parameters to balance between performance and reliability. The system dynamically adjusts these parameters based on workload characteristics to maintain atomicity while scaling across servers.
3Reliability
If a single-master stores all file metadata, then consistency is maintained, but storage capacity and performance are limited by the single machine's resources
Solution Approach 1:
The patent transitions from a single-dimension centralized metadata storage model to a multi-dimensional distributed architecture. Metadata is distributed across multiple servers in a hierarchical namespace structure, adding spatial distribution as a new dimension. This dimensional change allows the system to scale storage capacity linearly with the number of servers while maintaining consistency through the hierarchical namespace organization and distributed consensus protocols.
Data Source
AI summary
A computer-implemented distributed file-system in a distributed data network in which metadata related to the files and directories of the file-system is distributed. A unique and non-reusable inode number is assigned to each file/directory of the file-system. A key-value store built up in rows is created for the distributed metadata. Each of the rows has a composite row key and a row value (key-value pair) where the composite row key for each file/directory includes the inode number of the parent directory, and a name of the file/directory. For files below the maximum file size, the entire file or portion thereof is encoded in the corresponding row value of the key-value pair. In this case, the corresponding composite row key holds the inode number of the file itself and an offset information of the data of the file in the row value. Files above maximum file size are stored in a large-scale storage.