Memory Controller Sparse Hash Deduplication Small Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication methods face challenges in efficiently detecting data similarity for small files and random IOs, particularly in contexts where small files are difficult to group into large association blocks and where prediction filters struggle with efficient content removal.
Innovation Solution
A memory controller that uses a probabilistic method to determine sparse hashes for incoming data, populating a sparse index and selecting representatives based on probability, thereby optimizing support for small files and random IOs by reducing the number of index accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deterministic methods are used to select representatives from fingerprint areas, then the sparse index can be populated efficiently, but the metadata overhead increases and access efficiency decreases for small files and random IOs
Solution Approach 1:
The patent changes the parameter selection method from deterministic to probabilistic. Instead of always selecting the same representatives (deterministic), the system now selects representatives based on probability distributions, allowing the sparse index to adapt to different data patterns including small files and random IOs, thereby reducing metadata overhead while maintaining deduplication effectiveness
Solution Approach 2:
The patent introduces dynamic representative selection where the system can adaptively choose different representatives based on current data characteristics. This dynamic approach allows the sparse index to be more efficient for small files and random IO operations, as the representative selection can be optimized based on the specific access patterns and file sizes encountered
2Ease of operation
If small files are handled individually through sparse index access, then each file can be processed, but the number of index accesses increases significantly reducing performance
Solution Approach 1:
The patent merges multiple small files into larger association blocks for processing. By combining small files that share common characteristics or access patterns into single association blocks, the system reduces the number of sparse index accesses required, thereby improving access efficiency while still maintaining the ability to handle small files effectively
3Reliability
If prediction filters are used to increase similarity detection probability, then successful detection rate improves, but the filter cannot efficiently remove referenced content requiring periodic zeroing
Solution Approach 1:
The patent changes the filter operation mode by introducing probabilistic parameter selection. Instead of using fixed deterministic parameters in the prediction filter, the system uses probabilistic parameters that allow the filter to naturally expire and be replaced without requiring periodic zeroing operations, thus maintaining high detection accuracy while eliminating maintenance overhead
Data Source
AI summary
A memory controller is for receiving incoming data and storing and/or transmitting the incoming data utilizing deduplication. The memory controller determines a second hash corresponding to one or more first hashes. The second hash is a sparse hash and the first hashes are strong hashes. The memory controller populates a sparse index with the second hash. The sparse index associates the second hash with the corresponding first hashes. The memory controller determines the second hash based on a probability that the second hash corresponds to a plurality of first hashes using a probabilistic method.