Metadata Index Segmentation for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed storage systems face challenges in efficiently managing metadata indexes that grow excessively large, exceeding the capacity of volatile storage, which affects read and write operations and requires a solution to reduce index size while ensuring timely data retrieval.
Innovation Solution
The method involves archiving single transaction write operations and optimizing read operations by selecting an incrementally lower index from the metadata index to retrieve data objects, and dynamically deleting segment index numbers and offsets when the index size exceeds a threshold, allowing for a reduced metadata index that can be stored in non-volatile storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the metadata index stores all segment index numbers to enable direct data retrieval, then data retrieval speed is improved, but the index size grows excessively large exceeding volatile storage capacity
Solution Approach 1:
The patent segments the metadata index into two parts: a reduced metadata index stored in volatile storage containing only frequently accessed segment index numbers, and a full metadata index stored in non-volatile storage containing all segment index numbers. This segmentation allows the system to maintain fast retrieval for common operations while preserving complete indexing capability in non-volatile storage.
Solution Approach 2:
The patent extracts only the necessary portion of the metadata index (frequently accessed segment index numbers) into volatile storage, leaving the complete index in non-volatile storage. This extraction enables the system to work with a manageable subset of the full index during active operations, resolving the contradiction between retrieval speed and storage capacity.
2Device complexity
If a reduced metadata index is stored in volatile storage to reduce overhead, then resource overhead is minimized, but direct retrieval of all data objects becomes more complex
Solution Approach 1:
The patent introduces a mapping mechanism that acts as an intermediary between the reduced metadata index in volatile storage and the full metadata index in non-volatile storage. When a segment index number is not found in the reduced index, the system uses the mapping mechanism to locate it in the full index, thereby maintaining ease of operation while reducing resource overhead.
Solution Approach 2:
The system performs preliminary actions by pre-loading frequently accessed segment index numbers into the reduced metadata index during system initialization or periodic updates. This preliminary action ensures that common retrieval operations can be performed quickly without requiring access to the full metadata index, simplifying the retrieval process for the majority of cases.
3Reliability
If the complete metadata index is maintained in non-volatile storage, then all data objects can be retrieved, but the system cannot leverage fast volatile storage for indexing
Solution Approach 1:
The patent segments the metadata index storage into volatile and non-volatile portions, allowing the system to maintain retrieval completeness by keeping the full index in non-volatile storage while utilizing volatile storage for frequently accessed entries. This segmentation enables the system to leverage fast volatile storage for indexing operations without sacrificing the ability to retrieve all data objects.
Solution Approach 2:
The patent implements a dynamic metadata index management system where the reduced metadata index in volatile storage is periodically updated or reloaded from the full metadata index in non-volatile storage based on access patterns. This dynamic approach allows the system to adapt to changing data access requirements while maintaining both retrieval completeness and storage efficiency.
Data Source
AI summary
According to an example method, a read request associated with a second data object is received, where the second data object is identifiable by a second object identifier. A second segment index number is determined that would identify a location of the second data object is absent from a metadata index. In response to determining that the second segment index number is absent from the metadata index, an incrementally lower index is selected in the metadata index, where the incrementally lower index is a first segment index number that identifies a location of the first data object. The second data object is retrieved using the first segment index number and a first offset corresponding to the first segment index number.