Aggregated Object Storage With Manifest Metadata Batching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently handling both large and small data objects without degrading performance, particularly when combining different storage approaches that improve efficiency for one type of data can negatively impact another.
Innovation Solution
A system and method that utilizes parallel processing to aggregate data objects and metadata, storing them in data blocks and metadata storage equipment, with mechanisms to identify and manage duplicate objects, and employing batch operations to reduce transaction overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional storage systems store each data object individually with separate metadata transactions, then data retrieval is straightforward, but transaction overhead increases significantly when storing multiple small data objects
Solution Approach 1:
The patent combines multiple small data objects into a single aggregated data block, storing them together with shared metadata. This merging approach reduces the number of individual storage transactions from N (for N data objects) to just 1 aggregated transaction, dramatically reducing transaction overhead while maintaining efficient retrieval through the manifest file that maps individual object offsets within the aggregated block.
2Loss of time
If the system aggregates multiple data objects into a single storage unit, then transaction overhead is reduced, but metadata processing complexity increases
Solution Approach 1:
The patent segments the metadata into two distinct components: a manifest file that contains offsets and metadata for each individual data object within the aggregated block, and a shared metadata structure that applies to the entire aggregated block. This segmentation allows the system to benefit from aggregated storage while maintaining simple, organized metadata that can be processed efficiently.
3Productivity
If different storage approaches are used for large and small data objects, then performance for each type is optimized, but system complexity increases
Solution Approach 1:
The patent implements a universal aggregated storage mechanism that can handle data objects of any size through the same manifest-file-based approach. The system uses a general-purpose data block structure with a manifest that can accommodate varying numbers and sizes of data objects, eliminating the need for separate storage paths for different data sizes while maintaining optimized performance through batched aggregation.
Data Source
AI summary
The technologies described herein are generally directed toward writing multiple data objects to object storage. An example method can include receiving an aggregated data object comprising a set of data objects and metadata corresponding to respective data objects of the set of data objects. The method can further include updating the metadata to comprise the respective storage locations. Further, the method can include updating the metadata to include the respective storage locations. The method can further include allocating the metadata to be stored by metadata processing equipment, resulting in the metadata being stored in metadata storage equipment.