Ordered Metadata Structure for Container Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed storage systems, grouping small data objects into container objects leads to inefficiencies, increased processing power usage, metadata overhead, and potential data integrity issues, particularly during operations like repair and compaction.
Innovation Solution
An ordered metadata structure is implemented, where each data object is assigned a successively incremented identifier, and each container object is assigned an ordered identifier based on the highest data object identifier it contains, with a metadata structure maintaining entries for each container object including these identifiers and references to their data, enabling efficient access and merging of container objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If small data objects are stored individually in a distributed storage system, then data object management is simple, but storage efficiency decreases and IOPS requirements increase
Solution Approach 1:
The patent merges multiple small data objects into container objects, where each container object groups related data objects together. This reduces the number of individual metadata entries needed while maintaining the ability to manage and retrieve individual data objects within the container, thereby improving storage efficiency without proportionally increasing metadata overhead.
2Productivity
If container objects are used to group data objects, then storage efficiency improves, but tracking metadata becomes complex and data integrity may be compromised
Solution Approach 1:
The patent implements a nested metadata structure where container objects contain references to data objects, and the system maintains nested levels of metadata tracking. This nested approach allows the system to track both individual data objects and their container relationships, ensuring data integrity while maintaining storage efficiency. The metadata structure preserves the hierarchical relationship between containers and their contents.
3Speed
If conventional metadata structures are used for container objects, then implementation is straightforward, but access speed decreases during compaction operations
Solution Approach 1:
The patent pre-organizes metadata structures before compaction operations are needed. By maintaining pre-sorted and pre-indexed metadata that reflects the current state of container objects, the system eliminates the need for complex real-time reorganization during compaction. This preliminary organization of metadata enables fast access speeds even during dynamic operations like compaction, as the metadata is already in an optimized state for retrieval.
Data Source
AI summary
An ordered data object identifier denoted by a value is assigned to each data object grouped to a container object, wherein data object identifiers with successively incremented values are assigned to successive data objects. An ordered container identifier is assigned to each container object with the greatest value of the identifiers of the data objects grouped thereto. A metadata structure with an entry for each container objects is stored. Each entry comprises the ordered container identifier and a reference to the corresponding data. The metadata structure is ordered according to the values of the container identifiers. A request to read a data object contains a corresponding data object identifier. It is determined to which container object the data object is grouped, by locating the first container identifier in the metadata structure with a value greater than or equal to that the requested data object.