Subspace Hash Index Partitioning for Memory-Constrained Deduplication

The patent divides the global hash index into multiple subspace indexes based on data characteristics (e.g., file type, access pattern, age). Each subspace index can be independently managed and loaded into memory only when needed, rather than loading the entire index at once. This segmentation allows the system to maintain good index serving performance for active subspaces while keeping memory consumption manageable.

The patent implements dynamic loading and unloading of subspace indexes based on workload demands. The system can adaptively load relevant subspace indexes into memory during high-demand periods and unload them when memory pressure increases or demand decreases. This dynamic approach allows the system to optimize the balance between index serving performance and memory resource consumption in real-time.

The patent partitions the data dataset into multiple subspaces based on various criteria such as file type, access frequency, and temporal characteristics. Deduplication operations are then performed independently within each subspace rather than across the entire dataset at once. This segmentation reduces the computational complexity from O(n) for the whole dataset to O(n/k) for each of k subspaces, significantly reducing computational time while still achieving effective deduplication ratios within each subspace.

The patent applies deduplication selectively to specific subspaces based on their characteristics and deduplication potential. Rather than uniformly applying deduplication to all data, the system identifies subspaces with high deduplication potential (e.g., backup data, log files) and applies intensive deduplication there, while using lighter approaches for subspaces with low deduplication potential. This partial action approach achieves significant storage savings in critical areas without the full computational overhead of universal deduplication.