File Clustering via Bloom Filter Bitmaps for Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating an optimal dendrogram for file systems with millions of files is computationally intensive and impractical due to the need to calculate the similarity of all possible file pairs, making it difficult to identify clusters that maximize cohesion and minimize differences.
Innovation Solution
The cluster definition process estimates similarities using bitmaps represented as bloom filters, which approximate the optimal dendrogram by sampling unique segments and focusing on the K smallest offsets, reducing computational complexity while maintaining sufficient granularity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all possible file pair similarities are calculated to generate an optimal dendrogram, then clustering accuracy is improved, but computational complexity increases exponentially
Solution Approach 1:
The patent segments the similarity calculation process by focusing only on the K smallest offsets for each file rather than calculating all possible file pair similarities. This segmentation reduces the computational space from O(N²) to O(NK), where N is the number of files and K is a small constant, thereby resolving the contradiction between clustering accuracy and computational complexity
Solution Approach 2:
The patent extracts only the essential information needed for clustering by using bitmaps to represent file segments and focusing solely on the K smallest offsets. This extraction eliminates unnecessary computational overhead while preserving the core similarity information needed for accurate clustering
2Reliability
If complete dendrogram generation is performed for millions of files, then optimal clustering is achieved, but processing time becomes impractical
Solution Approach 1:
The patent applies partial action by computing only the necessary portion of the dendrogram structure - specifically, it calculates similarities based on K smallest offsets rather than complete file pair comparisons. This partial computation achieves sufficient clustering quality for practical purposes while reducing processing time from exponential to linear complexity
Solution Approach 2:
The patent performs preliminary action by pre-computing bitmaps for each file that encode the positions of their segments. These pre-computed bitmaps enable efficient similarity estimation without requiring repeated full file comparisons, thereby reducing processing time while maintaining clustering optimality
Data Source
AI summary
Clustering files in deduplication systems is based on an estimate of similarity between files in a file system. The estimates of similarity are based on how much content the files share, where the estimate of how much content is shared is based on an estimate of segments shared. The estimate of segments shared is based on segment offsets found in the files' bitmap vectors of segment offsets. The found segment offsets are used to generate a cluster definition approximating an optimal data structure for clustering files that share content. The approximated optimal data structure defines clusters hierarchically arranged based on the offset numbers of the found segment offsets.