Word-Aware Content-Defined Chunking for Text Analytics

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional deduplication processes split words across segments, which hampers the performance of text analytics and maintains larger logical data sizes compared to deduplicated data.

Innovation Solution

Implementing a word-aware content-defined chunking method that respects word boundaries by searching for segment boundaries within defined data ranges and using a window to identify candidate offsets, prioritizing whitespace positions as segment boundaries.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If conventional deduplication processes are used to reduce storage capacity requirements, then storage efficiency is improved, but text analytics performance deteriorates because words are split across segments

Engineering Contradiction:

Improvestorage capacity requirementsVSAvoidtext analytics performance

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The data is segmented into chunks using content-defined chunking that identifies segment boundaries at whitespace positions rather than arbitrary byte positions. This segmentation approach maintains word integrity across segments while still achieving deduplication benefits, resolving the contradiction between storage efficiency and text analytics performance

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The chunking process applies different boundary detection logic at different positions in the data - specifically prioritizing whitespace positions as segment boundaries. This local quality adjustment ensures that segments align with natural linguistic boundaries, preserving word integrity for text analytics while maintaining deduplication effectiveness

Inventive Principle:

Principle #3Local quality

2Quantity of substance

If segment boundaries are placed at arbitrary byte positions for deduplication, then deduplication ratios are improved, but word integrity deteriorates causing words to be split across segments

Engineering Contradiction:

Improvededuplication ratiosVSAvoidword integrity

Core Design Contradiction:

Quantity of substanceVSManufacturing precision

Solution Approach 1:

The system performs preliminary scanning of the data to identify whitespace positions before finalizing segment boundaries. This preliminary action ensures that segments are predetermined to align with word boundaries, preventing word splitting while maintaining effective deduplication ratios

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The segment boundary definition is changed from arbitrary byte positions to whitespace-based positions. This parameter change in the chunking algorithm fundamentally resolves the contradiction by making segment boundaries dependent on linguistic structure rather than fixed positional parameters, thereby preserving word integrity

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS12265513B2Word aware content defined chunking

Publication Date: 2025.04.01 EMC IP HLDG CO LLC

AI summary

One example method includes, in a data buffer that includes one or more words and whitespaces, calculating a hash value of data in a window that is movable within the data buffer, comparing the hash value to a mask, and when the hash value matches the mask, identifying a position of the window in the data buffer as a chunk anchor position, searching for a whitespace nearest the chunk anchor position, and designating an offset of the whitespace as a segment boundary.