Metadata Fragmentation in Distributed Encoded Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed encoded storage systems face challenges in efficiently storing and retrieving data due to the high cost and reduced durability of metadata storage on expensive, low-latency solid state disks (SSDs), which are used to maintain metadata responsiveness and redundancy.
Innovation Solution
The solution involves encoding metadata into fragments and storing these fragments on backend storage elements, while using low-latency frontend storage elements only for metadata fragment identifiers, allowing quick retrieval and maintaining the same redundancy level as the encoded data fragments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If metadata is stored on expensive SSDs to ensure high responsiveness, then system responsiveness is improved, but storage cost increases
Solution Approach 1:
The patent segments metadata into multiple fragments and distributes them across backend storage elements. Only fragment identifiers (not the full metadata) are stored on frontend SSDs, reducing the quantity of expensive storage required while maintaining fast access capability through the fragmented structure.
Solution Approach 2:
The patent extracts the essential identifying information (fragment identifiers) from the complete metadata and stores only these extracted elements on expensive frontend SSDs. The full metadata fragments are removed from frontend storage and placed on cheaper backend storage, significantly reducing SSD usage while preserving system responsiveness.
2Quantity of substance
If metadata is stored on backend storage elements with data, then storage cost is reduced, but system responsiveness deteriorates
Solution Approach 1:
Metadata is segmented into fragments distributed on backend storage, but the critical fragment identifiers remain on fast frontend storage. This segmentation allows the system to use slower backend storage for bulk metadata while maintaining fast access paths through the identifier cache on frontend SSDs.
Solution Approach 2:
The patent applies local quality by storing different types of information in different locations: fragment identifiers (small, frequently accessed) are stored on fast frontend SSDs, while full metadata fragments (large, less frequently accessed) are stored on slower backend storage. Each location optimizes for its specific access patterns.
3Reliability
If metadata is duplicated across multiple datacenters for redundancy, then reliability is improved, but storage cost and complexity increase
Solution Approach 1:
The patent segments metadata into fragments that can be independently distributed and replicated across datacenters. This segmentation enables more efficient redundancy strategies where not every fragment needs to be replicated everywhere, reducing the total storage overhead for achieving reliability.
Solution Approach 2:
The fragment identifier structure serves multiple functions: it uniquely identifies metadata fragments, enables fast lookup on frontend storage, and facilitates distributed storage and replication across multiple datacenters. This multi-functionality reduces the need for separate redundancy mechanisms.
Data Source
AI summary
A data object can be encoded into a plurality of encoded data fragments and stored on backend storage elements in a distributed encoded storage system. The identifiers and metadata corresponding to each encoded fragment of the data object can be stored in a single metadata unit, which is stored on the backend as encoded fragments. The identifiers of the metadata fragments can be associated with the data object and stored on a low latency frontend storage device. Thus, the amount of metadata per data object stored on expensive low latency frontend storage is reduced to the fragment identifiers. The fragment identifiers can be quickly retrieved, and used to retrieve the identifiers and metadata corresponding to the encoded data fragments from the backend, for retrieval of the data object itself.