File Deduplication System for Heterogeneous Cloud Compatibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
File systems capable of data reduction struggle to maintain deduplication characteristics when recovering raw files for backup or transfer to heterogeneous systems or clouds, limiting their applicability.
Innovation Solution
A file deduplication processing system that includes a processor and storage device, which loads a file dedup subsystem to deduplicate raw files into vault files and descriptors, allowing partial recovery of data chunks for updating and re-deduplication, optimizing file processing while maintaining data reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If files are recovered as raw files before backup or transfer to heterogeneous file systems, then compatibility with other machines and clouds is improved, but deduplication characteristics are lost and storage efficiency deteriorates
Solution Approach 1:
The patent segments files into fixed-size data chunks and stores them in a chunk store with deduplication. Each chunk is independently hashed and stored only once if duplicates are found. This segmentation allows the system to maintain deduplication characteristics while providing chunk-level access for heterogeneous system compatibility.
Solution Approach 2:
The patent introduces a file conversion layer that acts as an intermediary between the deduplicated chunk store and heterogeneous file systems. This layer converts chunk sequences into raw file formats when needed for backup or transfer, maintaining deduplication in the storage layer while ensuring compatibility at the interface layer.
2Reliability
If all data chunks in vault files are read and recovered during write operations, then data integrity is improved, but processing time and I/O operations increase
Solution Approach 1:
The patent extracts only the specific data chunks that need to be updated during write operations, rather than reading and recovering all chunks in vault files. The system identifies affected chunks through hashing and selectively retrieves them, maintaining data integrity while minimizing I/O operations and processing time.
Solution Approach 2:
The patent applies partial action by reading only the necessary portion of data chunks required for write operations. Instead of recovering all chunks in vault files, the system performs selective chunk retrieval based on the specific write operation requirements, reducing unnecessary I/O overhead while ensuring data integrity for the affected portions.
Data Source
AI summary
A file deduplication processing system is provided. The system deduplicates raw files to generate deduplicated vault files and a descriptor indicating a storage location of each data chunk in the vault files corresponding to the raw files. When receiving a writing request of a write data, the system finds at least one data chunk including old data corresponding to the write data according to the descriptor, loads and recovers the data chunk whose boundary is not overlapped with a boundary of the write data in the vault file comprising the old data corresponding to the write data so as to generate an update data by incorporating the recovered data chunk and the write data, deduplicates the update data to generate a new vault file and stores the same in the chunk store, and updates a content corresponding to each data chunk in the descriptor.