Two-Tier Storage Metadata Compression via Dynamic Namespace Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional file system interfaces are inadequate for trillion-scale object stores due to limitations such as a limited set of inodes, making them inoperable and causing metadata bottlenecks in storage systems, which affect performance and capacity.
Innovation Solution
A two-tier distributed storage system is implemented, with a global single namespace comprising dynamically-loadable namespaces in the capacity tier of object storage, allowing only a dynamic subset of metadata in the fast tier for improved usability and scalability, balancing memory and storage costs while maintaining low latency and high performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional file system interfaces are used to manage metadata in trillion-scale object stores, then the system maintains simplicity and ease of operation, but the system becomes inoperable due to limited inodes and metadata bottlenecks
Solution Approach 1:
The patent divides the namespace into dynamically-loadable segments that can be loaded into memory as needed. Instead of maintaining a single large metadata structure in memory, the system segments the namespace into smaller units that can be dynamically loaded and unloaded, allowing the system to handle trillion-scale object stores while keeping in-memory metadata manageable.
Solution Approach 2:
The patent implements dynamic loading and unloading of namespace segments based on access patterns and system state. The metadata structure transitions from static to dynamic, allowing the system to adapt memory usage to actual workload requirements, thereby scaling to trillion-scale object stores without permanent metadata bottlenecks.
2Speed
If all metadata is kept in the fast tier for quick access, then the system achieves low latency and high performance, but the memory cost becomes prohibitively expensive
Solution Approach 1:
The patent applies local quality by keeping frequently accessed metadata in the fast tier and less frequently accessed metadata in the capacity tier. Each namespace segment has different characteristics based on its access pattern, and the system optimizes the placement of each segment individually rather than treating all metadata uniformly, thereby reducing overall memory requirements while maintaining performance for hot data.
Solution Approach 2:
The system dynamically adjusts which metadata resides in the fast tier versus the capacity tier based on access patterns. Namespace segments can be promoted from capacity tier to fast tier when accessed frequently and demoted when accessed less frequently, allowing the system to optimize the balance between access speed and memory usage adaptively.
3Device complexity
If a limited set of inodes is used in traditional file systems, then the device complexity is reduced, but the system becomes inoperable for trillion-scale object stores
Solution Approach 1:
The patent segments the inode space into multiple namespace segments, each with its own inode allocation. Instead of using a single large inode table that would require trillions of entries, the system divides the namespace into smaller segments that can be independently managed and loaded, effectively providing unlimited inode capacity while maintaining manageable segment sizes.
Solution Approach 2:
The patent introduces a new dimension to inode management by adding the namespace segment hierarchy. Instead of a flat inode space, the system organizes inodes within namespace segments that can be dynamically loaded, creating a hierarchical structure that extends the effective inode capacity beyond traditional limitations while keeping individual segment complexities manageable.
Data Source
AI summary
Example embodiments of the present invention relate to a method, a system, and a computer program product for managing metadata. The method includes determining a common characteristic of metadata entries for a plurality of files in a directory of a file system and computing a representative metadata entry for the plurality of metadata entries according to the common characteristic of the metadata entries. The plurality of metadata entries in the file system then may be represented according to the computed representative metadata entry.