Selective Inline Data Deduplication via File Type Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication techniques face challenges in efficiently processing data inline, particularly with files that are well-compressed or have inherent redundancy, leading to delays and increased storage capacity needs, which is unsuitable for inline deduplication.
Innovation Solution
The proposed solution involves a data deduplication apparatus with a processor and memory that uses a deduplication engine, hash algorithms, and definition files to identify and remove duplicates, allowing for selective deduplication based on file type and format, enabling efficient inline processing by determining the suitability of data for deduplication before storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication processing is applied to all data including well-compressed files, then storage capacity is improved, but processing time and CPU usage increase significantly
Solution Approach 1:
The patent applies different quality levels of deduplication processing to different data types. Well-compressed files (binary executables, compressed archives) are identified and excluded from inline deduplication, while text-based files are processed. This local differentiation of processing quality resolves the contradiction by applying intensive processing only where beneficial.
Solution Approach 2:
The system performs preliminary identification and classification of files before applying deduplication processing. By pre-identifying well-compressed files through file type detection and compression ratio estimation, the system avoids wasting processing time on files that cannot be effectively deduplicated, thus resolving the time-capacity tradeoff.
2Quantity of substance
If inline deduplication processing is applied to well-compressed files, then storage capacity is improved, but system performance and usability deteriorate
Solution Approach 1:
The system implements local quality control by applying high-performance deduplication only to text-based files where linear scanning and hashing are efficient, while excluding binary and compressed files from inline processing. This resolves the contradiction by matching processing intensity to data characteristics.
Solution Approach 2:
The system performs preliminary file type classification and compression assessment before inline deduplication. Files are pre-screened to identify those suitable for fast deduplication processing, ensuring that system resources are dedicated only to files where inline processing will be productive.
3Quantity of substance
If all files are processed through deduplication, then deduplication ratio is improved, but processing complexity increases
Solution Approach 1:
The patent implements local quality differentiation by applying simple file type-based filtering to binary/compressed files and more sophisticated compression-ratio-based filtering to text files. This tiered approach achieves high deduplication ratios where possible while maintaining manageable processing complexity through rule-based classification.
Data Source
AI summary
Data is selectively deduplicated such that portions of data suitable for deduplication are passed to a deduplication engine (1064) and stored in a first store (1072).