Hierarchical Namespace Service with Distributed Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems face challenges in providing hierarchical namespace support, particularly when transitioning from on-premises storage with hierarchical directory structures to cloud storage systems using flat namespaces, as they struggle to maintain compatibility and efficiency in data access and management.
Innovation Solution
A hierarchical namespace service is implemented, which enables commands referencing file system objects using hierarchical namespace identifiers to be executed in a flat namespace by mapping these identifiers to corresponding flat namespace identifiers, allowing for efficient execution and management of file system operations without physical data movement, and dynamically scales name resolution nodes based on directory activity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If cloud storage systems use a flat namespace, then data access efficiency and system scalability are improved, but compatibility with hierarchical directory structures required by big data applications is lost
Solution Approach 1:
The patent introduces a namespace translation service as an intermediary layer between applications requiring hierarchical namespaces and the underlying flat namespace storage system. This service translates hierarchical namespace identifiers to flat namespace identifiers, enabling applications to interact with the flat namespace as if it were hierarchical, thus maintaining compatibility while preserving the efficiency benefits of flat namespace storage
Solution Approach 2:
The namespace translation service is divided into multiple distributed translation nodes that independently handle different portions of the namespace translation workload. Each node maintains a local cache of namespace mappings and can process translation requests autonomously, improving both scalability and compatibility simultaneously
2Adaptability or versatility
If namespace translation service is implemented, then compatibility with hierarchical namespaces is achieved, but system complexity increases due to additional translation layers and components
Solution Approach 1:
The translation nodes automatically discover and register new namespace mappings through a publish-subscribe mechanism. When new files or directories are created in the flat namespace, the system automatically publishes namespace information to relevant translation nodes, which then self-update their local caches without requiring manual configuration or complex centralized management
Solution Approach 2:
Instead of maintaining a single centralized namespace translation table, the patent implements distributed copies of the translation service across multiple nodes. Each node maintains a local copy of the namespace mapping cache, eliminating single points of failure and reducing the complexity of centralized namespace management while maintaining hierarchical namespace compatibility
3Speed
If namespace translations are cached, then translation speed and efficiency are improved, but data consistency may be compromised when namespace mappings change
Solution Approach 1:
The system implements a feedback mechanism where translation nodes subscribe to namespace change events. When namespace mappings change in the flat namespace, the system automatically publishes update notifications to relevant translation nodes, which then invalidate their cached entries and request fresh translations, ensuring data consistency while maintaining fast translation speeds through caching
Solution Approach 2:
Translation nodes periodically refresh their namespace mapping caches by re-querying the authoritative namespace information. This periodic refresh ensures that cached translations remain consistent with the current state of the flat namespace, balancing the need for fast translations with data consistency requirements
Data Source
Figure 1~2
Figure 3~5
Figure 6~7
AI summary
A service enables a command that refers to a file system object using a hierarchical namespace identifier to be executed against the file system object in a flat namespace. The service selectively distributes the command to one of a plurality of name resolution nodes based on a directory name included in the hierarchical namespace identifier. The identified node resolves the directory name to a flat namespace identifier that is used to execute the command against the flat namespace. After communicating with at least one storage node to resolve a directory name, each name resolution node stores a mapping of the directory name to the corresponding flat namespace identifier in a cache, so that subsequent resolutions of that directory name may be performed more efficiently. Cache entries may be invalidated when an operation occurs that impacts the relevant mapping and/or based on system considerations such as cache expiry.