File System Deduplication Using Directory Timestamp Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems face inefficiencies in identifying files for deduplication, as scanning all files can be costly and time-prohibitive, and require extensive computing resources.
Innovation Solution
Implementing a method that marks directories with logical timestamps to skip subtrees during searches, and organizing metadata to preserve locality and order of chunks, allowing for faster identification and processing of files for deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all files are scanned to identify files for deduplication, then complete deduplication coverage is achieved, but processing time and computing resources increase significantly
Solution Approach 1:
The patent applies preliminary action by maintaining timestamp metadata that is updated in advance whenever files are created or modified. This pre-computed timestamp information allows the deduplication system to quickly identify candidate files without performing a complete scan, thus reducing processing time while maintaining deduplication coverage.
Solution Approach 2:
The patent extracts only the necessary subset of files for deduplication by using timestamp metadata to filter out files that have not changed. Instead of processing all files, the system identifies and processes only those files whose timestamps indicate they have been created or modified since the last deduplication operation.
2Reliability
If all files are scanned to identify files for deduplication, then complete deduplication coverage is achieved, but computing resources are excessively consumed
Solution Approach 1:
The patent extracts only the necessary subset of files for deduplication by using timestamp metadata to filter out files that have not changed. Instead of processing all files, the system identifies and processes only those files whose timestamps indicate they have been created or modified since the last deduplication operation.
Solution Approach 2:
The patent applies partial action by performing deduplication only on the subset of files that have changed, rather than scanning and processing all files in the storage system. This reduces computing resource consumption while maintaining effective deduplication coverage for modified files.
3Adaptability or versatility
If metadata is organized without preserving locality, then storage flexibility is improved, but read and write operation speed decreases
Solution Approach 1:
The patent applies local quality by organizing metadata so that related information (such as timestamps and file identifiers for nearby files or files in the same directory) is stored close together. This preserves locality of reference, enabling faster read operations when processing files sequentially or in related groups, while still maintaining storage flexibility through the structured metadata format.
Data Source
AI summary
The disclosure provides techniques for deduplicating files. The techniques include, upon creating or modifying a file, placing a logical timestamp of the current logical time, within a queue associated with the directory of the file. The techniques further include placing the logical timestamp within a queue of each parent directory of the directory of the file. To determine a set of files for deduplication, the techniques disclosed herein identify files that have been modified within a logical time range. The set of files modified within a logical time is identified by traversing directories of a storage system, the directories being organized within a tree structure. If a directory's queue does not contain a timestamp that is within the logical time range, then all child directories can be skipped over for further processing, such that no files within the child directories end up being within the set of files for deduplication.