Edge Storage Deduplication via Difference Portion Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file virtualization techniques, such as network deduplication, are ineffective in reducing network communication amounts between hybrid cloud and edge-core systems, especially when dealing with updated chunks that have significant stubified areas or small update sizes, leading to increased network communication.
Innovation Solution
A storage device with a processor that determines whether to transmit the difference portion or its fingerprint based on the size of the difference portion and the fingerprint size, optimizing data transfer by choosing the most efficient method between direct data transfer and deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If network deduplication is performed by calculating FP of chunks in edge and transferring to core, then duplication determination can be performed in core, but network communication amount increases when most of chunk is stubified area or update area is small
Solution Approach 1:
The patent segments the chunk into stubified area and non-stubified area, and further divides the non-stubified area into difference portions and same portions. By processing only the difference portions through FP calculation and transmission, rather than the entire chunk, the system reduces network communication while maintaining deduplication accuracy for the actual modified data.
Solution Approach 2:
The patent extracts only the necessary difference portions from the chunk for FP calculation and transmission to the core storage. This extraction approach eliminates the need to transmit the entire chunk or even the stubified portions, significantly reducing network communication overhead while preserving the ability to perform accurate deduplication on the actual changes.
2Reliability
If FP of chunk is calculated and transmitted to core storage, then deduplication can be performed, but communication amount increases when update area in chunk is small
Solution Approach 1:
The patent applies partial action by calculating FP and transmitting only the difference portions that actually changed, rather than the entire chunk. This partial processing approach is sufficient for deduplication purposes since only the difference portions need to be compared with existing data in the core storage, thereby reducing communication overhead when update areas are small.
Data Source
AI summary
Provided is an edge storage having a central processing unit (CPU). The edge storage is connected to another storage device capable of deduplicating and managing a data of a file in units of a predetermined chunk via a network, in which, when a data of a newly created file or an updated file is transferred to the other storage device, the CPU determines whether to transmit a difference portion to a core storage or to transmit a finger print of a difference chunk to the core storage, based on a size of the difference portion and a size of the finger print of the difference chunk which is a chunk including the difference portion and transmits the difference portion or the finger print to the core storage according to a determination result.