Erasure-Coded Object Manifests for In-Place Insert and Overwrite
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems using erasure coding face inefficiencies when updating or modifying data, particularly in scenarios where data needs to be overwritten or inserted at points other than the end, due to the requirement of copying large amounts of data, which is slow and resource-intensive.
Innovation Solution
A storage cluster that allows for efficient editing and updating of objects using erasure coding by enabling flexible insertion, overwrite, and append operations without requiring data copying, utilizing a mechanism where each erasure set on disk can correspond to multiple manifest erasure sets, allowing for distinct encoding and representation of sparse or repetitive data, and enabling insertion at any point within the object.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional erasure coding is used for data modification, then data integrity and redundancy are maintained, but large amounts of data copying is required which slows down operations and consumes excessive memory and storage resources
Solution Approach 1:
The patent segments the erasure-coded object into multiple erasure sets, each representing a logical range of data. This segmentation allows modification operations to be performed on individual erasure sets or subsets thereof, rather than requiring copying and re-encoding of the entire object. The manifest tracks these segmented erasure sets, enabling selective updates that maintain data integrity while improving modification speed.
Solution Approach 2:
The patent applies local quality by allowing different portions of the object (different erasure sets) to have different properties and be treated independently. When modifying data, only the affected local portion (specific erasure set or range) needs to be updated, not the entire object. This localized approach reduces the amount of data copying required while maintaining overall data integrity through the erasure coding scheme.
2Reliability
If traditional erasure coding is used for data modification, then data redundancy is maintained, but large amounts of memory and storage are required for data copying
Solution Approach 1:
By segmenting the object into erasure sets and allowing independent modification of subsets, the patent reduces the quantity of data that needs to be held in memory and storage during modification operations. Instead of requiring full object copies, only the necessary erasure sets or ranges within them need to be accessed and updated, thereby reducing memory and storage resource consumption while maintaining redundancy.
Solution Approach 2:
The patent changes the parameter of data representation by introducing the concept of erasure sets as intermediate structures between the original data and the erasure-coded segments. This parameter change allows for more efficient manipulation during modification operations, reducing the need to hold large amounts of data in memory and storage while preserving the redundancy properties of erasure coding.
3Ease of operation
If flexible data insertion and overwrite operations are enabled, then ease of operation is improved, but device complexity increases due to the need for selective insertion and merging operations
Solution Approach 1:
The patent segments the data structure into erasure sets with clearly defined logical ranges, which simplifies the complexity of merge operations. By organizing data into these manageable segments, the system can perform selective insertion and overwrite operations on specific erasure sets or ranges within them, making the operations more straightforward despite the underlying complexity of erasure coding.
Solution Approach 2:
The manifest serves as an intermediary structure that tracks the mapping between logical data ranges and physical erasure sets. This intermediary simplifies the complexity of merge operations by providing a clear interface for specifying insertion and overwrite operations in terms of logical ranges, which the system then translates into the appropriate physical operations on erasure sets, thereby improving ease of operation without overwhelming complexity.
Data Source
AI summary
Segments of a local range can appear multiple times in the sequence of erasure sets in a manifest without the data being duplicated. A subset of the content of an erasure set can be included in the logical object. It is not required that all erasure sets in an object use the same encoding. A manifest can indicate one or more portions of the logical object for which no data has been provided, and for which a “fill” pattern is specified. A portion of data can be inserted at any point inside the object by inserting a reference to one or more segment sets containing that data into the manifest. If the desired insertion point is within an existing segment set, then that manifest set can be “split” by incorporating two references to it, with the first specifying the range of data before the insertion point, and the second specifying the range of data after the insertion point. Many operations are thus enabled on an erasure-coded object in Object Storage such as “append”, “insert,”“overwrite” and “merge.”