Fingerprint-Based Differential Storage to Avoid Read Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently managing large data volumes with limited storage capacity, leading to read amplification and reduced performance due to separate storage of reference and differential data, which complicates indexing and search processes.
Innovation Solution
Storing reference and differential data in the same read range of a storage system, using similar and deduplication fingerprints to determine data relationships, allowing for efficient data storage and reducing read amplification by reading both types of data with a single I/O operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If reference data and differential data are stored in discrete and distributed manner, then data can be stored separately, but read amplification is caused and read performance deteriorates
Solution Approach 1:
The patent merges reference data and differential data into the same storage unit, allowing both types of data to be stored together rather than in discrete locations. This combining eliminates the need for separate read operations, thus avoiding read amplification and improving read performance while maintaining the ability to store and manage different data types efficiently
2Device complexity
If reference data and differential data are stored separately, then data organization is simplified, but index structure and search process become more complex
Solution Approach 1:
The patent segments data into reference data and differential data components within the same storage unit, using fingerprint technology to identify and organize these segments. This segmentation allows for efficient indexing and search operations by enabling the system to quickly locate specific data portions without requiring complex external index structures
3Quantity of substance
If similar data deletion is implemented, then data capacity is reduced, but storage space utilization needs improvement
Solution Approach 1:
The patent changes the parameter of data representation by storing only differential data (the differences between similar datasets) rather than complete copies. This parameter change enables significant reduction in storage space requirements while maintaining the ability to reconstruct original data when needed, thus improving storage space utilization without losing data capacity
Data Source
Figure 1~2
Figure 3
Figure 4~6
AI summary
A data storage method in a storage system and a related device are provided. The method includes: calculating a similar fingerprint of first to-be-stored data to obtain a first similar fingerprint (S1101), where the first similar fingerprint is used to determine whether the first to-be-stored data is similar to stored data; determining reference data based on the first similar fingerprint, where a similar fingerprint of the reference data is the first similar fingerprint (S1102); determining first differential data between the to-be-stored data and the reference data based on the reference data, where the reference data is stored in a first storage unit (S1103); and storing the first differential data in a second storage unit, where the first storage unit and the second storage unit belong to a read range of a same read IO (S1104). According to the method, read amplification can be avoided and read performance can be improved.