Block Storage Deduplication Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional deduplication methods become inefficient when dealing with highly intelligent backup applications that identify uniquely changed blocks, leading to increased computing costs despite potential savings.
Innovation Solution
Implementing a system with optimized block storage for change block tracking that uses a 'no deduplication' indicator to avoid unnecessary deduplication processing, generating a global unique identifier (GUID) for new data blocks instead of calculating hashes, and segmenting data without intensive methods to store files efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional deduplication methods are used on backup systems that already identify changed blocks, then storage space is reduced, but computing cost increases significantly
Solution Approach 1:
The system applies deduplication processing selectively rather than universally. When the backup application sends data with a 'no deduplication' indicator (meaning the data is already identified as uniquely changed), the system skips the hash calculation and deduplication processing, accepting that some storage efficiency may be lost in exchange for avoiding excessive computing costs. This partial application of deduplication resolves the contradiction by balancing storage optimization with computational efficiency.
2Quantity of substance
If deduplication processing is performed on all incoming data, then duplicate blocks are eliminated, but processing time increases
Solution Approach 1:
The system performs deduplication processing only on data blocks that do not have the 'no deduplication' indicator. For data already identified as uniquely changed by the backup application, the system skips processing entirely, reducing processing time while accepting that no additional duplicate elimination occurs on that data. This selective approach resolves the time-cost contradiction.
Solution Approach 2:
The backup application performs preliminary identification of changed blocks before sending data to the storage system. By the time data arrives at the deduplicating storage system, the backup application has already determined which blocks are newly changed and marked them accordingly. This preliminary action eliminates the need for the storage system to perform time-consuming analysis on all incoming data, resolving the processing time contradiction.
3Productivity
If intelligent backup applications identify changed blocks locally, then data transfer is optimized, but deduplication efficiency decreases
Solution Approach 1:
The system accepts reduced deduplication efficiency on data marked with 'no deduplication' indicator in exchange for maintaining high data transfer efficiency. The backup application's local identification of changed blocks enables efficient data transfer by sending only necessary data, and the storage system preserves this efficiency by skipping deduplication processing on marked data, thus resolving the contradiction between transfer efficiency and deduplication efficiency.
Data Source
AI summary
A system for backup comprises an input interface configured to receive a request to back up a file. The system for backup additionally comprises a processor configured to determine whether the file comprises an indicator to back up without deduplication, and in the event the file comprises an indicator to back up without deduplication, process the file for deduplication storage without deduplication.