Remote Differential Compression Using Jaccard Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing remote differential compression (RDC) and data deduplication methods are ineffective for both high and low entropy data types, as they fail to efficiently identify and transfer only non-redundant data segments across wireless networks, leading to increased costs and computational overhead.
Innovation Solution
The method employs Jaccard segmentation to partition files into segments characterized by segment borders, uniqueness metrics, secure fingerprints, and hash functions, followed by identity-based and similarity-based deduplication to identify and transmit only unique segments, enabling efficient file replication across devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional RDC and data deduplication methods are used, then file replication can be performed, but redundant data is still transferred resulting in increased data transfer costs and computational overhead
Solution Approach 1:
The patent applies segmentation by dividing the target file into multiple segments using Jaccard segmentation before deduplication. This allows the system to identify and transfer only non-redundant portions of each segment, rather than processing the entire file as one unit, thereby reducing overall data transfer costs and improving replication efficiency.
Solution Approach 2:
The patent extracts and identifies redundant segments by comparing segmented target files with seed files using deduplication techniques. By taking out only the non-redundant segments for transfer, the system eliminates wasted data transmission while maintaining complete file replication capability.
2Productivity
If RDC methods are applied to identify and transfer only differences, then data transfer efficiency improves, but the methods fail to work effectively for both high entropy and low entropy data types
Solution Approach 1:
The patent achieves universality by developing a deduplication system that can effectively process both high entropy and low entropy data types through the combination of Jaccard segmentation and fingerprint-based comparison. This multi-functional approach allows the same methodology to work across diverse data types, overcoming the limitation of traditional RDC methods that are data-type specific.
Solution Approach 2:
The patent applies parameter changes by using Jaccard segmentation with specific parameters (segment size, overlap) that can be adjusted based on data characteristics. This allows the system to adapt its processing parameters to handle different entropy levels effectively, maintaining high transfer efficiency across various data types.
3Loss of energy
If redundant parts of data are identified and only non-redundant parts are transferred, then data transfer costs reduce, but computational overhead increases due to segmentation and comparison processes
Solution Approach 1:
The patent applies preliminary action by performing Jaccard segmentation and generating fingerprints of segments before the actual data transfer. By preparing the segmented structure and identification markers in advance, the system enables faster comparison and deduplication during transfer, reducing the computational burden during the critical transmission phase.
Data Source
AI summary
There is provided a method and apparatus for remote differential compression (RDC) and data deduplication. According to embodiments, when a sending device acquires a new target file, the following steps are performed. Initially, Jaccard segmentation is performed, followed by performing identity-based segment deduplication and similarity-based segment deduplication. The transmission of the target file in the deduplicated form to the recipient device is subsequently performed. The recipient device can then rebuild the original target file from the deduplicated form thus replicating the target file at the recipient device with the target file originally present at the sending device.