File Deduplication Candidate Classification via Event Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication storage systems require significant time and resources to analyze files for redundant data patterns, leading to inefficiencies in identifying files suitable for deduplication.
Innovation Solution
A method and system for classifying files as candidates for deduplication by detecting events suggestive of duplication, such as copying or sharing, and assigning attributes to indicate redundancy, allowing for targeted deduplication processes like source-side or target-side deduplication based on file characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional deduplication techniques are used to analyze all files for redundant data patterns, then deduplication can be performed, but considerable time and resources are consumed
Solution Approach 1:
The system performs preliminary classification of files into candidate groups based on heuristics (file type, size, modification time, access patterns) before actual deduplication analysis. This preliminary action filters out files unlikely to contain duplicates, reducing the scope of intensive analysis to only promising candidates.
Solution Approach 2:
The deduplication process is segmented into multiple stages: (1) preliminary classification using heuristics, (2) targeted analysis of candidate files, and (3) actual deduplication. This segmentation allows the system to perform quick filtering before investing resources in thorough analysis, thereby reducing overall time consumption while maintaining accuracy.
2Reliability
If traditional deduplication techniques analyze all files thoroughly, then complete deduplication identification is achieved, but considerable computing resources are consumed
Solution Approach 1:
The system performs preliminary classification of files into candidate groups based on heuristics (file type, size, modification time, access patterns) before actual deduplication analysis. This preliminary action filters out files unlikely to contain duplicates, reducing the scope of intensive analysis to only promising candidates.
Solution Approach 2:
Different levels of analysis are applied to different file groups. Files classified as high-probability candidates receive thorough analysis, while files classified as low-probability candidates receive minimal or no analysis. This local quality approach optimizes resource allocation based on individual file characteristics.
3Productivity
If files are classified as candidates for deduplication based on events detected, then processing time is reduced, but the complexity of detection and classification increases
Solution Approach 1:
The system leverages existing file system events and metadata (creation time, modification time, access patterns, copy operations) that are already being tracked by the operating system. Rather than implementing completely new detection mechanisms, the system utilizes self-service data from the file system itself, reducing the added complexity burden.
Solution Approach 2:
The detection and classification mechanism is designed to work with multiple types of files and operations universally. A single classification system handles various file types, copy operations, move operations, and access patterns, reducing the need for separate specialized detection mechanisms for each case.
Data Source
AI summary
A computer-implemented method may include identifying at least one file and detecting an event that is suggestive of at least a portion of the file being duplicated in at least one additional file. The computer-implemented method may also include classifying the file as a candidate for deduplication in response to detecting the event. The computer-implemented method may further include maintaining the file's candidate-for-deduplication classification for use in prompting a determination on whether the portion of the file is already stored within a storage device.