Dual Layer Deduplication for Application-Specific File Types
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional block-based deduplication methods in information processing systems are inefficient due to their inability to effectively deduplicate data at a byte level, leading to redundant storage and wasted resources, especially when minor changes in data cause misalignment and prevent deduplication of identical files.
Innovation Solution
Implementing a dual layer deduplication process that divides files into chunks based on their structure and type, allowing for byte-aligned deduplication by padding chunks to a consistent block size and using application-aware chunking algorithms to improve deduplication ratios without impacting system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If block aligned deduplication is used in block-based storage systems, then system performance is maintained, but deduplication effectiveness deteriorates due to inability to deduplicate data at byte level granularity
Solution Approach 1:
The patent divides files into smaller chunks at byte-level granularity before processing through block-based deduplication. This segmentation allows fine-grained deduplication of identical data portions while maintaining compatibility with block-based storage systems, thereby improving deduplication effectiveness without requiring complete system redesign
Solution Approach 2:
The patent implements a nested deduplication approach where byte-level chunking is nested within block-level deduplication processes. The file is first divided into byte-aligned chunks, which are then processed through the existing block-based deduplication infrastructure, creating a multi-layered deduplication system that combines both approaches
2Quantity of substance
If block level granularity deduplication is used, then storage system compatibility is maintained, but storage capacity utilization deteriorates due to retention of deduplicatable data
Solution Approach 1:
The patent performs preliminary byte-level chunking and hashing of file data before it enters the block-based storage system. By pre-processing the data to identify and mark duplicate chunks at the byte level, the system can achieve better storage capacity utilization while the actual deduplication operation remains compatible with block-level processing speeds
3Quantity of substance
If byte aligned deduplication is implemented, then deduplication ratio improves, but system performance deteriorates due to increased processing overhead
Solution Approach 1:
The patent introduces an intermediary layer that translates between byte-level chunk identifiers and block-level storage operations. This intermediary component allows the system to achieve byte-aligned deduplication ratios while interfacing with block-based storage systems, thereby maintaining system performance by avoiding direct byte-level I/O operations
Data Source
AI summary
An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The at least one processing device is configured to obtain a file and to determine a type of the file. A structure of the file may be determined based at least in part on the determined type of the file and at least one location in the file may be identified based at least in part on the determined structure. The file may be divided at the identified at least one location into a plurality of chunks and the plurality of chunks may be provided to a block deduplication module of a storage system where the block deduplication module is configured to perform a deduplication process based at least in part on the plurality of chunks.