Duplicate Database Chunking for Higher Data Compression Match Rates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Content-Defined Chunking (CDC) variable block algorithm's large granularity variable blocks decrease compression efficiency due to reduced matching probability of subsequent data fragments with the updated duplicate database.
Innovation Solution
A method that calculates fingerprints and digests of data fragments, compares them with existing variable blocks, and updates the duplicate database with smaller granularity variable blocks by adding sub-fragments when differences are found, increasing the likelihood of matching subsequent data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a CDC variable block algorithm uses large granularity variable blocks, then the data block size is reduced, but the matching probability of subsequent data fragments with the duplicate database decreases
Solution Approach 1:
The patent segments the duplicate database into multiple sub-databases (first duplicate database, second duplicate database, etc.), each storing variable blocks of different granularities. When a data fragment is matched, the system not only stores the original large granularity variable block but also generates and stores smaller granularity sub-blocks (first sub-variable block, second sub-variable block, etc.) in separate sub-databases. This segmentation allows subsequent matches to find relevant blocks at appropriate granularity levels, resolving the contradiction between reducing block size and maintaining matching probability.
Solution Approach 2:
The patent changes the granularity parameter of stored variable blocks by generating multiple versions of the same data block at different granularity levels. The system stores both the original large granularity variable block and generated smaller granularity sub-blocks, each with different size parameters. This parameter transformation enables the system to adapt to different matching scenarios, allowing large blocks for initial matches and smaller blocks for subsequent precise matches, thus resolving the contradiction between block size reduction and matching probability maintenance.
2Productivity
If the duplicate database is updated with a new variable block using delimitation sliding window mechanism, then the data structure is updated, but the compression efficiency decreases due to reduced matching probability
Solution Approach 1:
The patent performs preliminary action by proactively generating and storing smaller granularity sub-variable blocks when creating or updating variable blocks in the duplicate database. Instead of waiting for subsequent matches to fail, the system anticipates future matching needs by pre-computing and storing alternative granularity versions of data blocks. This preliminary preparation ensures that when subsequent data fragments arrive, there are already appropriate-sized blocks available for matching, thereby maintaining compression efficiency without sacrificing processing speed.
Solution Approach 2:
The patent creates copies of the original variable block at different granularity levels. When a variable block is stored in the duplicate database, the system generates copies with smaller granularities (first sub-variable block, second sub-variable block, etc.) and stores them in separate sub-databases. These copied versions at reduced granularities increase the likelihood of matching subsequent data fragments without requiring re-processing of the original large blocks, thus maintaining compression efficiency while preserving processing speed.
Data Source
AI summary
Embodiments of the present invention provide a data processing method and a data processing device. If to-be-compressed data includes a data fragment that is the same as a first half of a variable block in a duplicate database and is different from a second half of the variable block, a new variable block with granularity smaller than that of the matching variable block can be generated, and the new variable block is added to the duplicate database. The granularity of the new variable block is smaller, which increases the probability of subsequent to-be-compressed data matching the updated duplicate database, thereby improving compression efficiency.