File Deduplication Excluding Archival Schedules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication processes often waste processing resources by performing deduplication on files that will soon be archived, as they do not consider the schedule of archival jobs.
Innovation Solution
Implement a deduplication system that is aware of upcoming archival jobs by identifying files scheduled for archival within a defined timeframe and excluding them from the deduplication process, thereby conserving processing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deduplication is performed on all files without considering archival schedules, then deduplication completeness is improved, but processing resource waste increases
Solution Approach 1:
The system performs preliminary identification of files scheduled for archival within a defined time window before executing the deduplication operation. By proactively determining which files will be archived soon and excluding them from deduplication, the system avoids wasting processing resources on files that will be removed anyway, while still maintaining deduplication effectiveness on files that will remain in the system.
2Productivity
If files scheduled for archival are excluded from deduplication, then processing efficiency is improved, but deduplication coverage decreases
Solution Approach 1:
The system applies partial deduplication by selectively processing only those files that are not scheduled for archival in the near future. This partial action approach focuses computational resources on files that will actually benefit from deduplication, achieving sufficient coverage for files that remain in the system while avoiding unnecessary processing of files that will be archived soon.
Data Source
AI summary
A system can identify a first group of files that are scheduled to be modified by another operation within a next defined amount of time. The system can remove the first group of files from the files of the file system with respect to a deduplication operation to produce a second group of files. The system can perform a sampling phase on the second group of files, wherein the sampling phase identifies a first group of blocks, and wherein the first group of blocks corresponds to duplicate file system blocks. The system can identify a second group of blocks that correspond to the first group of files. The system can remove any blocks from the first group of blocks that have a respective duplicate in the second group of blocks to produce a third group of blocks. The system can perform the deduplication operation on the third group of blocks.