Dynamic Namespace Loading in Two-Tier Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional file system interfaces are inadequate for trillion-scale object stores due to limitations such as a limited set of inodes, making them inoperable and causing metadata bottlenecks in storage systems, which affect performance and capacity.
Innovation Solution
A two-tier distributed storage system is implemented, featuring a global single namespace with dynamically-loadable namespaces maintained in a capacity tier of object storage, where only metadata for active namespaces is loaded into a fast tier for improved usability and scalability, balancing memory and storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional file system interfaces are used to manage storage, then data access is simplified, but metadata overhead becomes excessive and performance degrades at trillion-scale
Solution Approach 1:
The patent segments the monolithic namespace into multiple isolated namespaces, each manageable independently. This allows the system to handle trillion-scale objects by dividing the metadata management into smaller, manageable units, reducing the metadata overhead that would otherwise cripple performance while maintaining simplified access through the unified namespace interface.
Solution Approach 2:
The patent extracts the namespace isolation mechanism from traditional file system interfaces, implementing it at the object store level. This extraction allows the system to maintain the simplicity of file system access while eliminating the metadata overhead problem by managing namespaces as separate, isolated entities within the object store architecture.
2Speed
If all namespaces are loaded into fast tier storage, then data access speed improves, but memory costs increase significantly
Solution Approach 1:
The patent implements dynamic namespace loading where namespaces are loaded into the fast tier storage only when needed and unloaded when not in use. This dynamic approach allows the system to maintain high access speeds for active namespaces while significantly reducing the total memory capacity required, as only a subset of namespaces resides in fast storage at any given time.
Solution Approach 2:
The patent employs periodic loading and unloading of namespaces based on access patterns. Namespaces that are frequently accessed are loaded into fast tier storage, while those that are not accessed are kept in slower, cheaper storage. This periodic action optimizes the balance between access speed and memory capacity requirements.
3Adaptability or versatility
If a unified namespace is maintained across all storage, then application portability is improved, but metadata management complexity increases
Solution Approach 1:
The patent introduces an intermediary layer that translates unified namespace operations into isolated namespace operations. This intermediary maintains application portability by presenting a consistent namespace interface to applications, while simultaneously managing the complexity of multiple isolated namespaces internally, thus reducing metadata management complexity for applications.
Solution Approach 2:
The patent creates isolated copies of namespace metadata for each namespace rather than maintaining a single unified metadata structure. This copying approach simplifies metadata management by allowing each namespace to be managed independently, reducing the overall complexity while maintaining application portability through the unified interface layer.
Data Source
AI summary
Example embodiments of the present invention relate to a method, a system, and a computer program product for maintaining dynamically loadable namespaces. The method includes presenting, to an application, a file system having a global single namespace (SN) comprising a plurality of dynamically-loadable namespaces (DLNs) maintained in a capacity tier (CT) of object storage of a storage system. In response to a job submitted by the application, dynamically loading a first dynamically-loadable namespace, including the metadata only for the first dynamically-loadable namespace, associated with the job from the capacity tier of object storage into a fast tier (FT) of storage of the storage system.