Deduplication Commonality Detection via Reference Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mechanisms for determining commonality in deduplicated data sets are inefficient and lack scalability, making it difficult to identify shared data segments across multiple files without expanding or decompressing data.
Innovation Solution
The solution involves generating and maintaining commonality information during deduplication using a datastore suitcase and filemaps, which include reference counts, indices, and offsets to track shared data segments, allowing for efficient detection of segment commonality without reading actual data, and using hash algorithms like MD5 or SHA-1 to identify unique IDs for each segment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented to reduce storage costs, then storage efficiency is improved, but the ability to detect commonality in the deduplicated data set becomes difficult
Solution Approach 1:
The system performs preliminary actions by generating and maintaining commonality information during the deduplication process itself. Reference counts, indices, and offsets are created and updated as data segments are processed, so that when commonality detection is needed, the information is already prepared and readily available without requiring additional data expansion or decomposition.
Solution Approach 2:
The patent introduces intermediary data structures including reference counts, indices, and offsets that mediate between the deduplicated data storage and the commonality detection function. These intermediaries track shared data segments across multiple files, enabling efficient detection without directly manipulating the actual data content.
2Difficulty of detecting and measuring
If existing commonality detection mechanisms are used, then some level of detection is possible, but scalability is limited
Solution Approach 1:
The system segments data into discrete data segments with unique identifiers, and maintains separate tracking structures (reference counts, indices, offsets) for each segment. This segmentation allows the commonality detection mechanism to scale by processing segments independently rather than analyzing entire files, enabling efficient detection across large data sets.
Solution Approach 2:
Instead of copying actual data content for comparison, the system creates and maintains copies of metadata information (reference counts, indices, offsets) that represent data segments. These metadata copies enable commonality detection without requiring manipulation of the actual data, significantly improving scalability.
3Measurement precision
If data is expanded or decompressed to determine commonality, then detection accuracy is improved, but processing time and resources increase
Solution Approach 1:
The patent extracts only the necessary information for commonality detection - namely reference counts, indices, and offsets - from the deduplicated data structures. By taking out just these metadata elements rather than expanding or decompressing actual data content, the system achieves accurate detection with minimal processing time and resources.
Data Source
AI summary
Mechanisms are provided for efficiently determining commonality in a deduplicated data set in a scalable manner regardless of the number of deduplicated files or the number of stored segments. Information is generated and maintained during deduplication to allow scalable and efficient determination of data segments shared in a particular file, other files sharing data segments included in a particular file, the number of files sharing a data segment, etc. Data need not be expanded or uncompressed. Deduplication processing can be validated and verified during commonality detection.