Bloom Filter Index for De-duplication I/O Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As de-duplicating storage systems grow in scale and utilize increasingly dense hard disk drives, the cost of random access I/O operations for segment index lookups becomes a significant issue, leading to inefficiencies in data retrieval and storage.
Innovation Solution
Implementing a hybrid, parallel architecture that uses bloom filters for smaller segment indexes to avoid unnecessary disk lookups, while performing parallel on-disk lookups for larger indexes, thereby reducing the number of I/O operations and memory resources required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a segment index is stored on disk to track all stored segments, then the system can maintain comprehensive de-duplication capability, but the cost of random access I/O operations increases significantly
Solution Approach 1:
The patent divides the segment index into multiple parts: a bloom filter index stored in memory for quick lookups, and the full segment index stored on disk. This segmentation allows the system to maintain comprehensive de-duplication capability while reducing I/O operations by first checking the in-memory bloom filter before accessing disk storage.
Solution Approach 2:
The bloom filter serves as an intermediary data structure between the incoming segment and the full segment index on disk. It provides a fast preliminary check that can eliminate the need for disk I/O operations when a segment is not present, thereby reducing overall I/O costs while maintaining de-duplication reliability.
2Measurement precision
If the segment index grows to cover all segments in large-scale systems, then de-duplication accuracy is maintained, but the number of random I/O operations increases
Solution Approach 1:
The system performs a preliminary check using the bloom filter stored in memory before accessing the full segment index on disk. This preliminary action filters out segments that are definitely not present, allowing the system to maintain high lookup accuracy while improving data retrieval efficiency by avoiding unnecessary disk I/O operations.
Solution Approach 2:
The bloom filter provides a partial check that covers all segments but with probabilistic accuracy. While it may produce false positives, it eliminates false negatives and reduces the need for full disk index scans, thereby maintaining sufficient lookup accuracy while significantly improving retrieval efficiency in large-scale systems.
3Volume of stationary object
If dense hard disk drives are used to increase storage capacity, then system scalability is improved, but the cost of random access I/O operations increases
Solution Approach 1:
The patent replaces mechanical disk access with an in-memory bloom filter check for preliminary segment lookup. This substitution eliminates the need for mechanical disk head movements for segments that are not present in the bloom filter, thereby reducing random access I/O costs while maintaining the scalability benefits of dense hard disk drives for actual data storage.
Data Source
AI summary
Techniques to perform segment index lookups are disclosed. In various embodiments, for each of one or more segment index entries included in a first on disk segment index a corresponding set of values is stored in a bloom filter. The bloom filter is used to determine prior to performing an on disk segment lookup of the segment index with respect to a given segment whether each location in the bloom filter that is associated with the given segment has been set to said corresponding set of values. An on disk lookup is performed in parallel of a second on disk segment index that is not included in said subset of on disk segment indexes each of which has associated therewith a corresponding bloom filter.