Content Defined Segmentation With Minimum Size Bound

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data deduplication techniques are sensitive to high-level partitions of data, leading to inconsistent segmentation results, reduced effectiveness in large-scale systems, and inability to support parallel calculation without compromising deduplication efficiency.

Innovation Solution

A new segmentation algorithm that applies data-dependent minimum and maximum size bounds on blocks, using rolling hash values and hierarchical segmentation conditions to produce consistent segmentations independent of high-level partitions, enabling parallel calculation and efficient storage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If content defined segmentation is used without minimum size bound, then segmentation flexibility is improved, but block size variance increases and deduplication efficiency decreases

Engineering Contradiction:

Improvesegmentation flexibilityVSAvoidblock size consistency

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The patent introduces a minimum size bound parameter to constrain the block size produced during content defined segmentation. This parameter change transforms the unbounded segmentation process into a bounded one, ensuring that blocks maintain a minimum size threshold while preserving the content-defined segmentation approach. The minimum size bound is applied dynamically during the segmentation process to filter out blocks that would be too small, thereby reducing block size variance and improving deduplication efficiency.

Inventive Principle:

Principle #35Parameter changes

2Productivity

If high-level data partitioning is used, then parallel calculation capability is improved, but segmentation consistency across partitions deteriorates

Engineering Contradiction:

Improveparallel calculation efficiencyVSAvoidsegmentation consistency

Core Design Contradiction:

ProductivityVSStability of the object's composition

Solution Approach 1:

The patent applies segmentation at multiple levels: first dividing data into high-level partitions for parallel processing, then performing content defined segmentation within each partition. The minimum size bound is applied consistently across all partitions to ensure that edge cases at partition boundaries are handled uniformly. This multi-level segmentation approach allows parallel calculation while maintaining segmentation consistency through the universal application of the minimum size bound constraint.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements a feedback mechanism where the minimum size bound is enforced during the segmentation process, and segments that violate this bound are rejected or adjusted. This feedback loop ensures that regardless of how data is partitioned at the high level, the final segments produced after applying the minimum size bound constraint will be consistent and valid, thereby maintaining segmentation consistency across parallel partitions.

Inventive Principle:

Principle #23Feedback

3Manufacturing precision

If minimum size bound is applied strictly, then block size consistency is improved, but segmentation positions may be discarded reducing processing throughput

Engineering Contradiction:

Improveblock size consistencyVSAvoidsegmentation processing throughput

Core Design Contradiction:

Manufacturing precisionVSProductivity

Solution Approach 1:

The patent applies partial action by selectively enforcing the minimum size bound only when necessary. During content defined segmentation, the algorithm attempts to create segments based on content boundaries, and only discards or adjusts segments that violate the minimum size bound. This partial enforcement approach maintains block size consistency for critical segments while allowing flexible segmentation where the minimum bound does not conflict with content-defined boundaries, thereby balancing precision with processing throughput.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS9268786B2Applying a minimum size bound on content defined segmentation of data

Publication Date: 2016.02.23 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Applying a content defined minimum size bound on blocks produced by content defined segmentation of data by calculating the size of the interval of data between a newly found candidate segmenting position and a last candidate segmenting position of same or higher hierarchy level, and then discarding the newly found candidate segmenting position if a size of an interval of data is lower than the minimum size bound, or retaining the newly found candidate segmenting position if the size of the interval of data is not lower than the minimum size bound or if there is no last candidate segmenting position of a same or higher hierarchy level as the newly found candidate segmenting position. When a last candidate segmenting position of a same or higher hierarchy level becomes available, the evaluation is reiterated to converge edge segmenting positions of the outputs of consecutive calculation units.