Variable Length Segment Maps for Cloud Object Store Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud object stores face inefficiencies in modifying, deleting, or creating data objects due to the need to fetch and rewrite entire data objects, especially when managing snapshots and deduplicated data.
Innovation Solution
The implementation of variable length encoded segment maps allows for efficient storage and manipulation of deduplicated objects by using segment maps to facilitate operations like hole punching, overwriting, copying, and truncating without requiring the reading or rewriting of entire data objects, thereby supporting synthetic full snapshots.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire data objects are fetched and rewritten for modification or deletion, then data integrity is maintained, but storage operation efficiency deteriorates
Solution Approach 1:
The patent segments data objects into fixed-size chunks and introduces segment maps that track which chunks belong to each object. This allows operations to be performed on individual chunks or ranges of chunks rather than entire objects, enabling efficient modification and deletion without fetching complete data objects while maintaining data integrity through the segment map structure.
Solution Approach 2:
The segment map serves as an intermediary data structure between the object store and the data objects. It contains metadata about chunk assignments and enables the system to perform operations on specific portions of data objects without directly accessing or manipulating the entire object data, thus improving operation efficiency while preserving data integrity through controlled access via the segment map.
2Reliability
If entire data objects are rewritten for snapshot operations, then snapshot consistency is ensured, but computational resources are wasted
Solution Approach 1:
The patent extracts only the necessary segment map metadata required for snapshot operations rather than processing entire data objects. By working with segment maps that contain references to data chunks, the system can create and manage snapshots by manipulating only the metadata structures, ensuring snapshot consistency while dramatically reducing computational resource consumption compared to rewriting complete objects.
3Reliability
If data objects are stored as complete units, then data integrity is maintained, but storage space efficiency deteriorates
Solution Approach 1:
The patent divides data objects into fixed-size chunks and uses segment maps to track which chunks belong to which objects. This segmentation enables efficient deduplication where identical chunks across different objects are stored only once, and the segment map maintains references to these shared chunks, thereby improving storage space utilization while preserving data integrity through the reference structure.
Solution Approach 2:
Instead of storing complete data objects, the patent stores unique data chunks once and uses the segment map to create virtual copies through references. Multiple objects can reference the same chunk data without duplicating the actual stored bytes, achieving space efficiency while maintaining data integrity through the reference-based access model managed by the segment map.
Data Source
AI summary
Methods, non-transitory computer readable media, and computing devices are disclosed that facilitate efficient storage operations using variable length encoded segment maps for deduplicated objects. A key name and elements of a segment list are received with a request to store an object. Each of the elements corresponds to a segment associated with the object and comprises a segment hash. A segment map is generated for the object. The segment map includes a header and entries. Each of the entries includes at least an expanded offset, one of the segment hashes, and a start offset. The segment map is stored in an object store, as associated with the key name, along with segment data included in at least one of the elements.