DAG Namespace Management for Distributed Storage Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file storage systems face challenges in maintaining high availability, durability, and scalability, particularly when dealing with large files and concurrent user access, as they often struggle with consistency models and network complexities in distributed systems.
Innovation Solution
A distributed file storage service is designed with a modular architecture, incorporating a storage subsystem, metadata subsystem, and access subsystem, utilizing replication techniques like erasure coding and consensus-based state machines to ensure sequential consistency and handle concurrent requests efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If industry-standard file system interfaces are used in distributed systems, then compatibility and ease of operation are improved, but consistency and reliability deteriorate due to asynchronous interactions and network partitions
Solution Approach 1:
The patent introduces a namespace service as an intermediary layer between clients and storage devices. This mediator manages namespace entries and handles consistency across distributed storage devices, allowing standard file system interfaces to be used while maintaining reliability through centralized namespace management and version control mechanisms.
2Productivity
If distributed storage systems are designed for high scalability and availability, then productivity and adaptability are improved, but device complexity and difficulty of detecting and measuring increase
Solution Approach 1:
The patent segments the storage system into distinct components: namespace service, storage devices, and clients. Each component has a specific responsibility - the namespace service manages metadata and namespace entries, while storage devices handle data storage. This segmentation allows the system to scale independently and reduces overall complexity by localizing functions.
Solution Approach 2:
The patent introduces a namespace dimension that sits above the physical storage dimension. Instead of managing complexity at the storage device level, the namespace service provides an abstracted view of storage resources, allowing clients to access distributed storage through logical namespace entries rather than dealing with underlying device complexity.
3Reliability
If namespace management is distributed across multiple nodes, then availability and scalability are improved, but maintaining consistency and handling concurrent requests becomes more difficult
Solution Approach 1:
The patent implements feedback mechanisms through version numbers and conditional operations on namespace entries. When clients modify namespace entries, version numbers are updated and can be used to verify consistency. This feedback allows the distributed namespace service to detect and handle concurrent modifications, ensuring consistency while maintaining availability across multiple nodes.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A directed acyclic graph (DAG) is generated to represent a namespace of a directory. In response to a request to create a new object with a specified name, a hash value bit sequence is computed for the name. A plurality of levels of the DAG are navigated using successive subsequences of the bit sequence to identify a candidate node for storing a new entry corresponding to the specified name. If the candidate node meets a split criterion, the new entry and at least a selected subset of entries of the candidate node's list of entries are distributed among a plurality of DAG nodes, including at least one new DAG node, using respective bit sequences obtained by applying the hash function for each distributed entry.