Metadata Retrieval Tree for Distributed Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed object storage systems, full enumeration for retrieving objects based on metadata is resource-intensive, consuming significant CPU and disk resources and resulting in long retrieval times due to the lack of a directory tree structure for precise enumeration.
Innovation Solution
A metadata retrieval tree is constructed with nodes representing different metadata types, allowing for targeted enumeration through linked lists and index-based retrieval, enabling efficient node addition, deletion, and modification, and reducing the need for full enumeration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full enumeration is used to retrieve objects based on metadata in distributed object storage system, then all objects can be retrieved, but CPU and disk resources are consumed excessively and retrieval time becomes very long
Solution Approach 1:
The patent segments the metadata retrieval process by constructing a metadata retrieval tree that divides all metadata types into hierarchical levels. Each node represents a metadata type and contains objects filtered by that metadata, enabling partial enumeration instead of full enumeration. This segmentation allows the system to retrieve only the necessary portions of metadata rather than processing the entire dataset.
Solution Approach 2:
The patent introduces a new dimensional structure (the metadata retrieval tree) that organizes metadata types in a hierarchical manner. This tree structure adds a layer of organization between the flat storage structure and the retrieval process, enabling navigation through metadata dimensions to achieve targeted enumeration without scanning all objects.
2Measurement precision
If full enumeration is used to list all objects with specified prefix, then all matching objects can be found, but retrieval time increases significantly
Solution Approach 1:
The patent performs preliminary action by pre-organizing objects into the metadata retrieval tree structure during the indexing phase. Each object is associated with multiple metadata types and positioned in the appropriate nodes, so that during retrieval, the system can directly navigate to relevant nodes without performing full enumeration. This preliminary organization enables fast targeted retrieval.
Solution Approach 2:
The patent extracts the metadata indexing and organization function from the general storage system, creating a dedicated metadata retrieval tree structure. This extracted structure handles the enumeration and retrieval operations independently, allowing the system to extract only the necessary metadata information without processing unrelated data.
3Device complexity
If no directory tree structure is used in distributed object storage system, then system structure is simplified, but precise enumeration capability is lost
Solution Approach 1:
The patent makes the metadata retrieval tree serve multiple functions: it acts as an indexing structure, an enumeration guide, and a retrieval path map. This multi-functional structure is built upon the existing flat storage architecture without requiring a complete redesign, thereby maintaining simplicity while adding precise enumeration capability.
Solution Approach 2:
The metadata retrieval tree serves as an intermediary structure between the flat storage layout and the retrieval operations. It mediates between the simple storage structure and the need for precise enumeration by providing a navigational layer that translates retrieval requests into targeted access patterns without changing the underlying storage architecture.
Data Source
AI summary
Disclosed are a method and apparatus for retrieving and enumerating object metadata in a distributed storage system. The method includes: defining and configuring metadata of objects, and counting the number N of types of the metadata; constructing a metadata retrieval tree according to the metadata and the number N of the types of the metadata, each Mth node being associated with an Mth linked list; configuring objects in the Mth linked lists, the objects conforms to all metadata in metadata arrays in the Mth nodes; and configuring an index based on the metadata, pointing to a specified target node via the metadata retrieval tree according to the index, determining a target linked list according to the target node, and providing enumerated objects by the target linked list.