Hash-Based Deduplication Boundary Control for Stable Segment Sizes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hash-based deduplication systems face challenges in controlling segment sizes, leading to sensitivity to small insertions and deletions, which affects the accuracy and efficiency of data storage by invalidating chunk boundaries.
Innovation Solution
The system controls segment sizes by setting boundaries using a sequence of hash functions and logical tests, partitioning segment sizes into three regions with defined limits, allowing for variable chunk lengths and reducing sensitivity to insertions and deletions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fixed segment boundaries are used in hash-based deduplication, then storage efficiency is improved through duplicate detection, but the system becomes highly sensitive to small insertions and deletions which invalidates chunk boundaries
Solution Approach 1:
The patent applies dynamics by transitioning from fixed segment boundaries to variable-length segments that dynamically adapt to data changes. The system uses a minimum segment length parameter instead of fixed boundaries, allowing segments to flexibly adjust their size based on insertion and deletion operations while maintaining deduplication effectiveness.
Solution Approach 2:
The patent changes the parameter of segment boundary definition from fixed positions to variable lengths defined by a minimum segment length parameter. This parameter change allows the system to control segment size distribution and reduce sensitivity to small data modifications while preserving deduplication accuracy for significant changes.
2Adaptability or versatility
If variable segment lengths are implemented to reduce sensitivity to insertions and deletions, then adaptability is improved, but segment size distribution becomes uncontrolled leading to efficiency losses
Solution Approach 1:
The patent implements feedback by monitoring segment size distribution and adjusting the minimum segment length parameter accordingly. The system observes the distribution of segment sizes and modifies the parameter to achieve an optimal balance between adaptability to data changes and maintenance of storage efficiency, creating a self-regulating mechanism.
Solution Approach 2:
The patent applies preliminary action by pre-defining a minimum segment length parameter before data processing begins. This preliminary parameter setting establishes control over segment size distribution from the outset, preventing excessive fragmentation while maintaining flexibility to handle insertions and deletions.
3Measurement precision
If small minimum segment lengths are used to capture fine-grained changes, then detection precision is improved, but the number of segments increases exponentially reducing storage efficiency
Solution Approach 1:
The patent changes the parameter from very small minimum segment lengths to an optimized minimum segment length that balances detection precision with storage efficiency. By carefully selecting this parameter, the system achieves sufficient sensitivity to detect meaningful changes while avoiding exponential segment proliferation that would degrade performance.
Data Source
AI summary
Segment sizes are controlled by setting the size of a segment boundary in a hash-based deduplication system. A subsequence of size K of a sequence of characters S is set. Segment boundaries are set by using the sequence of the decreasingly restrictive logical tests if one of the sequence of the decreasingly restrictive logical tests returns a true value when applied on the sequence of characters S.