Deduplicated Data Tiering via Segmented Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data tiering approaches often result in significant data duplication and substantial memory and processing requirements for transferring data between tiers.
Innovation Solution
A controller and worker architecture is implemented within a tiering microservice to transfer data segments to the appropriate tier while minimizing duplicate segments and memory requirements, using a deduplicated storage system and object storage in a cloud environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data tiering approaches are used, then data can be transferred between storage tiers, but significant data duplication occurs and substantial memory and processing requirements are imposed
Solution Approach 1:
The patent segments data into fixed-size chunks and further divides chunks into variable-sized segments with unique identifiers. This segmentation enables selective transfer of only necessary segments between storage tiers, avoiding duplication of entire data blocks and reducing memory requirements during transfer operations.
Solution Approach 2:
The system performs preliminary actions by maintaining metadata about data segments in the active tier before actual data transfer to cold tier. This preliminary metadata preparation allows the system to identify and transfer only missing segments, minimizing data duplication and reducing the memory and processing requirements during the actual transfer operation.
2Quantity of substance
If data is transferred between storage tiers, then storage cost optimization is achieved, but data movement and redundancy increase
Solution Approach 1:
The patent uses metadata copying instead of full data copying. Metadata about data segments is maintained in the active tier, allowing the system to track what data exists in cold tier without physically copying or moving the actual data segments. This approach minimizes data movement while maintaining storage cost efficiency.
Solution Approach 2:
The system changes the parameter of data representation from full data blocks to segmented data with unique identifiers and metadata. This parameter change enables the system to track and transfer only necessary data segments, reducing overall data movement while maintaining storage cost optimization benefits.
3Speed
If metadata access speed is improved, then fast access to data information is achieved, but storage system complexity increases
Solution Approach 1:
The patent introduces metadata as an intermediary layer between the active tier and cold tier. This metadata intermediary contains information about data segments without requiring complex storage structures, enabling fast metadata access while keeping the overall storage system relatively simple through the use of standardized data structures.
Data Source
AI summary
One example method includes identifying objects that each include one or more segments to be transferred from a source storage tier to a target storage tier, determining a total amount of data to be transferred, using a tiering controller to create worker nodes operable to transfer the segments to the target storage tier, where a number of worker nodes created is based on the amount of data, transferring, from the source storage tier to the target storage tier, only those segments of the objects not already present in the target storage tier, and the transferring of the segments is performed by the worker nodes, and for each of the objects, placing metadata associated with that object in a bucket.