Storage Data Unit Deduplication Using Bloom Filter Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in managing deduplicated data, leading to fragmentation of data chunks due to repeated deduplication, which disrupts spatial locality and wastes capacity.
Innovation Solution
A control method for storage systems that divides data into chunks, generates unique values, and uses bloom filters to determine the most relevant storage data unit for deduplication, storing at least one deduplicated chunk in consecutive memory addresses to prevent fragmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is divided into chunks and stored in storage data units without considering relevance, then deduplication capacity is improved, but spatial locality deteriorates causing fragmentation
Solution Approach 1:
The patent segments data into chunks and further organizes chunks into data units with specific structures (source data units and storage data units). This segmentation allows deduplication to operate at the chunk level while maintaining organizational structure at the data unit level, preventing fragmentation by keeping related chunks together in the same storage data unit.
Solution Approach 2:
The patent introduces a new dimension of organization by creating storage data units that group chunks based on relevance (Hamming similarity) rather than just sequential storage. This adds an organizational layer that maintains spatial locality while enabling deduplication, resolving the contradiction between capacity efficiency and spatial continuity.
2Quantity of substance
If deduplication is performed repeatedly on the same data, then storage capacity efficiency is improved, but fragmentation increases disrupting spatial locality
Solution Approach 1:
The patent performs preliminary organization of chunks into storage data units based on relevance before final storage. By calculating Hamming similarities and grouping related chunks together in advance, the system prevents fragmentation that would occur with repeated deduplication operations, maintaining spatial locality while achieving capacity efficiency.
Solution Approach 2:
The patent changes the organizational parameter from simple sequential storage to relevance-based grouping using Hamming similarity metrics. This parameter change allows the system to maintain spatial locality by keeping related data together, even as deduplication operations are performed repeatedly over time.
3Stability of the object's composition
If chunks are stored in consecutive memory addresses, then spatial locality is improved, but deduplication management complexity increases
Solution Approach 1:
The patent introduces storage data units as intermediary structures between chunks and physical storage locations. These data units act as containers that maintain spatial locality by grouping related chunks together, while the metadata within data units simplifies the management of deduplication relationships, reducing overall system complexity.
Solution Approach 2:
The storage data unit structure serves multiple functions simultaneously: it maintains spatial locality by grouping related chunks, enables efficient deduplication through metadata tracking, and simplifies retrieval operations. This multi-functionality reduces management complexity while preserving spatial continuity.
Data Source
AI summary
A control method includes dividing a source data unit among a plurality of source data units received from a host into a plurality of source chunks; generating a unique value for each of the plurality of source chunks; inserting the unique values into a source bloom filter of the source data unit; calculating a Hamming similarity with a storage bloom filter of each of the plurality of active storage data units for the source bloom filter; selecting an active storage data unit based on the highest Hamming similarity as a target data unit; classifying each of the plurality of source chunks as a first deduplicated chunk, an already deduplicated chunk, or a unique chunk using the unique values; and storing at least one of the first deduplicated chunks in the target data unit, such that spatial locality of each chunk may improve and fragmentation may be alleviated.