Targeted Data Chunking Skipping Unchanged Regions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current backup/restore applications process all data to generate variable length data chunks, leading to significant time consumption due to repeated checksum computations, especially as data object size increases, as they start processing from the beginning of the data buffer without considering existing chunk boundaries.
Innovation Solution
Implement targeted chunking by identifying and adjusting the sliding window's starting point based on expected chunk boundaries, allowing the data chunking routine to skip processing previously unchanged data, thereby reducing unnecessary computations and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the data chunking routine processes all data from the beginning of the data buffer, then complete data chunks are generated, but significant time is consumed due to repeated checksum computations on unchanged data
Solution Approach 1:
The system performs preliminary actions by identifying expected chunk boundaries before the data chunking routine processes the data buffer. By calculating where chunks are expected to occur based on previous backup information, the system can adjust the sliding window starting point to skip unchanged data regions, avoiding redundant checksum computations while maintaining accurate chunk boundary detection
Solution Approach 2:
The data buffer is segmented into regions based on expected chunk boundaries. The system divides the processing task into segments by identifying boundaries where chunks are likely to occur, allowing the sliding window to jump between segments and process only the necessary portions of data that contain actual changes, rather than processing the entire buffer sequentially
2Reliability
If the sliding window starts at the beginning of the data buffer, then all data is processed ensuring complete coverage, but unchanged data is processed repeatedly increasing computational overhead
Solution Approach 1:
The system applies partial action by processing only the necessary portions of the data buffer rather than the entire buffer. By using expected chunk boundaries to determine where processing should start and stop, the system performs just enough processing to capture all changed data while avoiding redundant computation on unchanged regions, balancing completeness with efficiency
3Measurement precision
If checksum computation is performed on all data bytes, then accurate chunk boundaries are identified, but processing time increases linearly with data object size
Solution Approach 1:
The system performs preliminary calculation of expected chunk boundaries using information from previous backups before the data chunking routine executes. This preliminary action allows the system to pre-determine where chunks are likely to occur, so that during the actual processing, checksum computations are only performed on data regions that contain changes, rather than computing checksums for all data bytes from scratch
Data Source
AI summary
Targeted chunking of data is described. A system identifies a length of a sliding window that a data chunking routine applies to a data buffer to create data chunks. The system identifies an expected chunk boundary in the data buffer. The system adjusts the expected chunk boundary, based on the length of the sliding window. The system enables the data chunking routine to start applying the sliding window at the adjusted expected chunk boundary in the data buffer instead of starting application of the sliding window at a beginning of the data buffer.