Locality-Based Stream Segmentation for Efficient Data Deduplication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data storage and transmission methods are inefficient due to the lack of intelligent segmentation of identical data sequences, leading to redundant storage and transmission, and the need for separate evaluation processes to satisfy minimum and maximum segment length constraints.

Innovation Solution

The method involves determining a segmentation window by computing values associated with candidate boundaries using hash functions or other distinguishing functions, selecting a boundary based on these values, and using a stack mechanism to efficiently track extreme values, allowing for efficient storage and transmission by referencing previously stored segments.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If simple segmentation is used without intelligent boundary selection, then segmentation process is simple, but storage and transmission efficiency deteriorates due to redundant data

Engineering Contradiction:

Improvestorage and transmission efficiencyVSAvoidsegmentation process complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent divides data streams into segments with intelligently selected boundaries to maximize identification of identical segments across different data sets. By segmenting data at optimal boundaries rather than fixed intervals, the system groups identical byte runs together, enabling efficient deduplication through reference storage instead of redundant storage of identical segments.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the parameter of segment boundary selection from fixed or simple delimiters to dynamically optimized positions based on data content analysis. By analyzing data patterns and selecting boundaries that align with identical sequences, the system transforms the segmentation process from a static structure to an adaptive one that responds to data characteristics, thereby improving deduplication efficiency.

Inventive Principle:

Principle #35Parameter changes

2Quantity of substance

If segment boundaries are selected to maximize identical data grouping, then storage efficiency improves, but the complexity of boundary selection process increases

Engineering Contradiction:

Improveredundant data storageVSAvoidboundary selection process

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent performs preliminary analysis of data streams to identify optimal segment boundaries before actual segmentation occurs. By pre-processing data to locate identical byte runs and determine boundary positions that maximize segment matching, the system eliminates the need for complex real-time boundary selection during segmentation, reducing overall process complexity while achieving optimal deduplication.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The segmentation system uses the data's own characteristics (identical byte sequences, patterns, and structures) to automatically determine optimal boundaries without requiring external complex algorithms. The data essentially selects its own segmentation points by exhibiting natural repetition patterns that the system detects and exploits, simplifying the boundary selection process while maximizing storage efficiency.

Inventive Principle:

Principle #25Self-service

3Manufacturing precision

If separate evaluation processes are added to satisfy minimum and maximum segment length constraints, then constraint compliance improves, but processing complexity and time increase

Engineering Contradiction:

Improvesegment length constraint complianceVSAvoidevaluation process complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent combines the boundary selection process with the constraint satisfaction process into a single integrated operation. By selecting segment boundaries that simultaneously maximize identical data grouping and satisfy minimum/maximum length constraints within the same algorithmic framework, the system eliminates the need for separate evaluation processes, reducing complexity while ensuring constraint compliance.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system incorporates constraint checks into the preliminary boundary selection phase rather than adding separate post-processing evaluation steps. By determining feasible boundary positions that satisfy length constraints during the initial segment identification process, the system ensures constraint compliance is built-in from the start, avoiding additional processing complexity and time.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS7504969B2Locality-based stream segmentation for data deduplication

Publication Date: 2009.03.17 EMC IP HLDG CO LLC

AI summary

Selecting a segment boundary comprises determining a segmentation window is determined and computing a plurality of values associated with candidate boundaries within the segmentation window. Selecting a segment boundary further comprises selecting one of the candidate boundaries based at least in part on a comparison between two or more of the computed values and determining a boundary within the segmentation window.