Storage System Data Division for Deduplication Rate
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems that use inline data division methods for deduplication face a low deduplication rate due to improper division points, especially during data backup processes where changes in data writing sequences and commit events lead to suboptimal data division, resulting in reduced efficiency and increased storage costs.
Innovation Solution
A storage system with a data buffer and a dividing unit that generates divided data based on a preset division standard, ensuring residual and continuing data are properly divided and stored, allowing for optimal deduplication by redividing data at optimal points without disrupting the data backup process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the inline data division method is used to perform data division in real time during writing, then the processing speed and responsiveness are improved, but the deduplication rate deteriorates due to improper division points caused by changes in data writing sequences and commit events
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing optimal division points based on data content characteristics before actual data writing occurs. The system analyzes data patterns in advance and creates a division plan that will be executed during inline processing, thus maintaining both real-time processing speed and optimal deduplication rates by avoiding dynamic decision-making during commit events
Solution Approach 2:
The system implements feedback mechanisms by continuously monitoring data writing sequences and commit events, then adjusting division point selections based on observed patterns. The feedback loop analyzes the impact of previous division decisions on deduplication rates and refines future division points to optimize both processing speed and deduplication performance dynamically
2Stability of the object's composition
If data division is executed after all data are written (post-process method), then data division stability is improved, but the disk I/O load increases because the number of times of I/O is three times that of the inline method
Solution Approach 1:
The patent applies preliminary action by pre-calculating division points based on data content characteristics before actual data writing occurs. The system analyzes data patterns in advance and creates a division plan that will be executed during inline processing, thus maintaining both real-time processing speed and optimal deduplication rates by avoiding dynamic decision-making during commit events
Solution Approach 2:
The patent segments the data division process into distinct phases: pre-calculation of division points based on content characteristics, inline execution of division decisions during data writing, and post-verification of deduplication effectiveness. This segmentation allows the system to perform computationally intensive analysis beforehand while maintaining low I/O load during actual data operations
Data Source
AI summary
For the purpose of suppressing decrease of a deduplication rate in a storage system, a storage system according to the present invention includes: a data buffer; a dividing unit configured to generate divided data by dividing data inputted into the data buffer; and a storage processing unit configured to store the divided data into a storage device while eliminating duplicate storage. The dividing unit is configured to generate the divided data by dividing the data inputted into the data buffer by a preset division standard based on the content of the data and also divide, by the division standard, connected data that residual data being left without being divided by the division standard and continuing data continuing to the residual data and being inputted in the data buffer are connected.