Multi-Engine Hashing for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication and compression techniques face challenges in optimizing hash size and compression size, leading to suboptimal results in identifying and removing duplicate data, which affects storage efficiency and compression effectiveness.
Innovation Solution
Decoupling hash size from compression size allows for smaller hash sizes, enabling more effective duplicate detection while using multiple hash engines with different block sizes and skip capabilities to improve deduplication and compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hash size is increased to improve deduplication accuracy, then duplicate detection capability is improved, but processing time and computational resources are increased
Solution Approach 1:
The patent divides the data into multiple blocks and processes them using multiple hash engines simultaneously. Each hash engine processes a specific block or portion of the data, allowing parallel computation that reduces overall processing time while maintaining comprehensive deduplication coverage through multiple hash sizes.
Solution Approach 2:
The patent employs multiple hash engines with different hash sizes (e.g., 4KB, 8KB, 16KB) to process the same data set. By varying the hash block size parameter across different engines, the system can identify duplicates at different granularities, improving deduplication accuracy without requiring a single large hash computation that would be time-consuming.
2Quantity of substance
If compression size is increased to improve compression ratio, then storage efficiency is improved, but processing complexity and resource consumption are increased
Solution Approach 1:
The patent segments the compression process into multiple passes, where each pass handles a specific block size. The system processes data in smaller chunks (4KB, 8KB, 16KB) sequentially, which simplifies the compression algorithm implementation and reduces the complexity of handling large data sets while achieving high compression ratios through cumulative compression of all blocks.
Solution Approach 2:
The patent applies compression at multiple different block sizes, effectively performing partial compression operations that can be combined. By compressing at smaller block sizes (4KB, 8KB) in addition to larger sizes (16KB), the system achieves better overall compression ratio without requiring a single complex large-block compression operation, thus reducing processing complexity.
3Measurement precision
If multiple hash engines with different block sizes are used to improve deduplication, then duplicate detection capability is improved, but system complexity is increased
Solution Approach 1:
The patent merges the outputs of multiple hash engines by collecting hash values from all engines and processing them through a unified deduplication algorithm. The system combines the results from different hash block sizes (4KB, 8KB, 16KB) into a single deduplication process, which identifies duplicates across all hash levels while managing system complexity through integrated processing rather than separate independent systems.
Solution Approach 2:
The patent implements a universal processing framework that handles multiple hash engines with different block sizes through a common architecture. The same processing pipeline accommodates various hash configurations, and the system uses a unified approach to manage and process data from all hash engines, reducing the need for separate specialized handling of each engine type.
4Productivity
If skip capability is increased to improve processing speed, then CPU efficiency is improved, but measurement precision for duplicate detection may be reduced
Solution Approach 1:
The patent segments the data processing into multiple passes with different skip strategies. In earlier passes, the system uses larger skip values to quickly process data and identify obvious duplicates, improving CPU efficiency. In later passes, the system uses smaller skip values or no skipping to perform more thorough duplicate detection, maintaining measurement precision. This segmented approach allows the system to balance speed and accuracy across different processing stages.
Data Source
AI summary
A system for calculating a fingerprint across a data set by identifying a data set to hash, the data set comprising a set of data blocks, generating, by a first hash engine, a first hash for each data block in the set of data blocks within the data set, and generating, by a second hash engine, a second hash for each data block in the set of data blocks within the data set.