Content-Defined Tree for Cross-Silo Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to efficiently deduplicate and manage large data objects due to inefficiencies in computing differences, handling insertions and deletions, and minimizing transmission costs, particularly in versioning and data storage across multiple databases.
Innovation Solution
The use of content-defined trees, which are trees of cryptographic hashes where each leaf node represents a data chunk and parent nodes are hashes of concatenated child node hashes, allows for efficient indexing, deduplication, and storage by grouping nodes based on rolling hashes, enabling efficient comparison and retrieval of data objects across databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing systems store complete copies of data objects for versioning, then version integrity is maintained, but storage utilization deteriorates
Solution Approach 1:
The patent segments data objects into fixed-size chunks and represents each version as a content-defined tree where leaf nodes are hashes of chunks and parent nodes are hashes of concatenated child hashes. This segmentation allows storing only the minimal necessary information for each version while maintaining integrity through the tree structure and hash relationships.
Solution Approach 2:
The content-defined tree structure implements a nested hierarchy where parent nodes contain hashes of their child nodes, forming a nested structure. This nesting allows efficient representation of version relationships where common chunks between versions can be shared through the hierarchical hash structure, reducing overall storage requirements while maintaining version integrity.
2Reliability
If existing systems upload entire objects to create new versions, then version completeness is ensured, but network utilization deteriorates
Solution Approach 1:
The patent extracts only the essential information needed to represent a version - specifically the content-defined tree structure with chunk hashes - rather than uploading complete data objects. This extraction allows creating new versions by transmitting minimal data (only the tree structure and new chunk hashes) while ensuring version completeness through the cryptographic hash relationships that verify data integrity.
3Productivity
If existing systems compute differences between data objects, then deduplication efficiency is improved, but access requirements worsen
Solution Approach 1:
The patent performs preliminary action by pre-computing the content-defined tree structure and chunk hashes when data objects are initially stored. This preliminary computation creates an indexed structure that enables efficient deduplication operations without requiring access to the original source data objects, as the tree structure and hash relationships are self-contained and can be compared independently.
4Adaptability or versatility
If existing systems handle insertions and deletions in data objects, then data flexibility is maintained, but operational complexity worsens
Solution Approach 1:
By segmenting data into fixed-size chunks and representing versions as content-defined trees, the patent simplifies handling insertions and deletions. Modifications only affect specific chunks and their corresponding tree nodes, rather than requiring manipulation of entire data objects. The hierarchical structure allows localized updates while maintaining overall data flexibility.
Data Source
AI summary
In some aspects, a computing system may generate a content-defined tree. A content-defined tree may be a tree of cryptographic hashes where each leaf is a hash of a chunk (e.g., data chunk) of a data object, and each parent node (e.g., interior node) is the hash of a concatenation of the hashes of the parent's children nodes. To create parent nodes for the leaf nodes, a computing system may group leaf nodes together based on a rolling hash (e.g., a rolling hash of the hashes of the leaf nodes) satisfying a condition. Each parent node may include a hash that represents the concatenation of the hashes of the leaf nodes that fall under the corresponding parent node.