Data Deduplication via Fingerprint Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication systems face inefficiencies due to the increased metadata overhead and processing time when dividing data files into a large number of small segments, which can lead to data fragmentation and reduced deduplication performance.
Innovation Solution
A system that generates fingerprints for data segments, such as using SHA-1 hash fingerprints, to identify unique segments, allowing for efficient comparison and storage, rather than transmitting entire segments, thereby reducing communication volume and time, and utilizing similarity group identifiers to route segments to appropriate deduplication services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is divided into a large number of small segments to improve deduplication identification, then more duplicates can be identified and storage space saved, but metadata size increases, processing time increases, and data fragmentation increases
Solution Approach 1:
The patent applies segmentation by dividing data into segments of a predetermined size (e.g., 4KB to 64KB) to enable efficient deduplication. This segmentation allows the system to process and compare data in manageable chunks, improving deduplication effectiveness while controlling metadata overhead through size constraints.
Solution Approach 2:
The patent changes the parameter of segment size to an optimal range (4KB to 64KB) to balance deduplication efficiency with metadata overhead. By establishing a minimum and maximum segment size, the system optimizes the trade-off between identifying more duplicates and reducing metadata complexity.
2Quantity of substance
If data is divided into a large number of small segments to improve deduplication identification, then more duplicates can be identified, but deduplication time increases due to processing large numbers of small segments
Solution Approach 1:
The patent segments data into predetermined size blocks to enable efficient parallel processing and comparison. This segmentation strategy allows the system to identify duplicates more effectively while managing processing time through controlled segment dimensions.
Solution Approach 2:
The patent uses partial action by processing only the necessary portions of data through fingerprinting and hashing of segments. Instead of processing entire data streams, the system focuses on extracting and comparing key segment features, reducing overall processing time while maintaining deduplication effectiveness.
3Quantity of substance
If data is divided into a large number of small segments to improve deduplication identification, then more duplicates can be identified, but data fragmentation increases
Solution Approach 1:
The patent applies segmentation with predetermined size constraints (4KB to 64KB) to balance deduplication benefits with fragmentation control. This structured segmentation approach ensures that while data is divided into manageable segments for processing, the fragmentation remains controlled within acceptable parameters.
Data Source
AI summary
An access object service receives a communication of a stream of fingerprints, corresponding to segments of a file, from a source of the file, and identifies sequential fingerprints in the stream of fingerprints as a group of fingerprints. The access object service identifies a similarity group identifier for the group of fingerprints. The access object service communicates the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service identifies fingerprints in the group of fingerprints which are missing from fingerprint storage. The deduplication service communicates the identified fingerprints to the access object service, which communicates a request for segments of the file, corresponding to the identified fingerprints, to the source of the file. The deduplication service receives a communication of the requested segments from the source of the file and stores the requested segments.