Bitmap-Based Deduplication Identification for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data storage systems, identifying files that cannot benefit from deduplication is computationally intensive due to the lack of efficient methods to determine which segments belong to multiple files, leading to inefficient storage utilization and increased costs.
Innovation Solution
Representing each file's unique segments in memory using bitmaps and employing a process that compares neighboring files to identify those with excessive differences, allowing for the identification of files that should be moved to less expensive storage tiers without deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If files are compared to determine deduplication benefit, then storage efficiency is improved, but computational complexity increases significantly
Solution Approach 1:
The patent segments files into fixed-size chunks and represents each file's chunks using a bitmap data structure. This segmentation allows efficient comparison by processing individual chunk sets rather than entire files, reducing computational complexity while maintaining storage efficiency detection capability.
Solution Approach 2:
The patent introduces an intermediary data structure (bitmap) that mediates between raw file data and deduplication analysis. The bitmap represents file chunks in a compact form that enables efficient set operations, serving as an intermediary that simplifies the complexity of direct file comparison while preserving the ability to determine deduplication benefit.
2Reliability
If all files are analyzed for deduplication similarity, then storage cost optimization is improved, but processing time increases
Solution Approach 1:
The patent applies partial action by analyzing only neighboring files in sorted order rather than all possible file pairs. This approach processes a subset of relevant comparisons (files with similar names or patterns) to identify deduplication candidates, reducing processing time while still achieving cost optimization for the most likely candidates.
Solution Approach 2:
The patent performs preliminary sorting of files by name before conducting deduplication analysis. This preliminary action groups potentially similar files together, so that when the actual comparison occurs, only relevant neighboring files need to be analyzed, significantly reducing the overall processing time required for storage cost optimization.
3Measurement precision
If segment-to-file mapping is maintained, then segment ownership detection is improved, but storage requirements increase prohibitively
Solution Approach 1:
The patent inverts the traditional mapping approach by maintaining file-to-chunk bitmaps rather than chunk-to-file mappings. This inversion allows the system to determine segment ownership by checking which files have a chunk's bit set, enabling precise segment ownership detection without requiring prohibitively large inverted index structures.
Solution Approach 2:
The patent changes the data structure parameters from dense chunk-to-file mappings to sparse file-to-chunk bitmaps. This parameter change represents segment ownership information in a more compact form, where each file's bitmap only sets bits for chunks it contains, dramatically reducing storage requirements while maintaining the ability to detect segment ownership through bitwise operations.
Data Source
AI summary
Identifying files that do not deduplicate well in a storage system with deduplication facilitates optimizing storage capacity by moving the identified files to less expensive storage without deduplication. Any set of files can be examined to remove files that are identified as files that do not deduplicate well. The process of identification includes arranging the files in a predefined order and using bitmap representations of the unique segments in the files to determine a count of different segments in neighboring next files compared to the previous files, and removing from deduplication any next files that exceed a difference threshold. The bitmap representations of the files allows the identification processes to be performed efficiently for large datasets. Any over-identification of files is minimized by repeating the identification processes on the set of files after arranging them in the reverse order.