Metadata Fragment Encoding for Distributed Storage Read Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed encoded storage systems face inefficiencies in read operations due to the high cost and reduced durability of separate metadata storage on fast, low-latency SSDs, and the latency introduced by retrieving encoded data fragments from slower backend storage elements.
Innovation Solution
Storing identifiers and metadata in a single metadata unit, which is encoded into multiple fragments and distributed across backend storage elements, allowing quick retrieval of fragment identifiers from a low-latency frontend storage device to initiate data object processing before downloading all encoded data fragments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If metadata is stored on separate fast SSDs, then responsiveness to metadata access is improved, but storage cost increases and durability is reduced
Solution Approach 1:
The patent combines metadata with encoded data fragments into a single storage unit, eliminating the need for separate metadata storage on expensive SSDs. This merging approach allows both data and metadata to be stored together on durable backend storage while maintaining efficient access patterns through the encoding scheme.
Solution Approach 2:
The encoded storage units serve multiple functions simultaneously: they store both the encoded data fragments and their associated metadata, provide redundancy through the encoding scheme, and enable efficient retrieval operations. This multi-functionality eliminates the need for specialized separate storage systems.
2Reliability
If all encoded data fragments are downloaded before processing, then data integrity is ensured, but read operation latency increases
Solution Approach 1:
The patent performs preliminary decoding of metadata from the encoded storage units before the actual data retrieval operation. By pre-processing the metadata to extract information about the encoded data fragments, the system can initiate data retrieval earlier and parallelize operations, reducing overall latency while maintaining integrity through the encoding scheme.
Solution Approach 2:
The patent segments the retrieval process into independent parallel operations: metadata decoding, fragment identification, and data fragment retrieval can occur concurrently. This segmentation allows the system to avoid sequential waiting and reduce total read operation time while ensuring data integrity through the distributed encoding.
3Loss of information
If a large amount of metadata is stored for each data object, then tracking and retrieval information is complete, but storage efficiency decreases
Solution Approach 1:
The patent extracts only the essential metadata information needed for retrieval and embedding it within the encoded data fragment structure. By taking out only the necessary tracking and retrieval information rather than storing complete redundant metadata, the system maintains metadata completeness while significantly reducing storage overhead.
Solution Approach 2:
The patent merges metadata with the encoded data fragments in a space-efficient manner, where the metadata is integrated into the same storage units as the data itself. This combining approach eliminates the need for separate large-scale metadata storage while preserving all necessary information for tracking and retrieval operations.
Data Source
AI summary
A data object can be encoded into multiple encoded data fragments and stored on the backend of a distributed encoded storage system. The identifiers and metadata corresponding to each fragment of the data object can be stored in a first section of a metadata unit, and the initial part of the data object in a second section. The metadata unit is encoded into multiple metadata fragments, which are stored on the backend. The identifiers of the metadata fragments can be associated with the data object and stored on a fast frontend storage device. In response to a request to access the data object, the identifiers are used to retrieve the metadata fragments from the backend, and decode the metadata unit. The initial part of the data object is retrieved from the metadata unit and transmitted to the requesting client application to begin processing the data object.