Native File Deduplication via Chunk Stubs and Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mechanisms for maintaining deduplication in native file formats are limited, leading to inefficient resource usage and storage challenges, as they often rely on individual file compression which is not effective and can lead to adverse effects when applications attempt to reorganize or backup deduplicated files.
Innovation Solution
The solution involves analyzing files to identify components suitable for deduplication, generating chunk identifiers using hash algorithms, and storing these in deduplication dictionaries and datastore suitcases, allowing for transparent access to deduplicated data through file stubs while maintaining the actual file components for efficient storage and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If individual file compression is used to save storage resources, then storage space is reduced, but compression efficiency is low and resource usage increases
Solution Approach 1:
The patent segments files into fixed-size chunks (e.g., 4KB) and processes each chunk independently. This segmentation enables efficient deduplication by comparing only relevant portions of files rather than compressing entire files individually, thereby improving compression efficiency while maintaining storage savings
Solution Approach 2:
The patent merges multiple file chunks into a unified deduplication storage structure where duplicate chunks are stored only once with reference counting. This combining approach achieves superior compression ratios compared to individual file compression while reducing overall system resource usage through shared storage
2Quantity of substance
If deduplication is implemented to reduce storage footprint, then storage efficiency improves, but application functionality may be affected
Solution Approach 1:
The patent introduces file stubs as intermediary objects that reference actual data chunks in the deduplication storage. These stubs maintain the native file format interface that applications expect, allowing seamless application functionality while the actual data is stored efficiently in deduplicated form, thus resolving the contradiction between storage efficiency and application adaptability
3Adaptability or versatility
If native file formats are used to maintain application compatibility, then application functionality is preserved, but deduplication capability is limited
Solution Approach 1:
The patent segments native file formats into fixed-size chunks that can be independently processed for deduplication. This segmentation allows the system to apply deduplication techniques to native files without requiring format conversion, thereby maintaining both application compatibility and enhanced deduplication capability simultaneously
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces storage footprints, saves resources, and ensures seamless application operations by maintaining deduplicated data in native file formats, allowing for efficient deduplication without affecting application functionality.
Implementation Method 1
generating chunk identifiers using hash algorithms
Data Source
AI summary
Mechanisms are provided to maintain deduplication data in native file formats. Files, including entities such as volumes and databases, are analyzed to identify components suitable for deduplication. These components suitable for deduplication are delineated into chunks and identifiers are generated for each of the chunks. The identifiers are used to reference the chunks in deduplication dictionaries that provide locations indicating where deduplicated chunks are stored. The components in the files are replaced with file handles or stubs that applications can use to access deduplicated data. Applications can continue to perform operations on the files as though no deduplication has occurred.