Distributed KV Store Garbage Collection With Integrity Checking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face inefficiencies in garbage collection and data integrity checking, leading to resource consumption, false positives, and delayed detection of data loss, with concurrent operations causing resource contention and performance degradation.
Innovation Solution
A streamlined garbage collection process that integrates data integrity checking, identifying unused block IDs and missing block IDs concurrently, reducing resource consumption and improving detection efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If probabilistic data structures (e.g., Bloom filters) are used to track data block usage for garbage collection, then transmission efficiency is improved due to compact representation, but false positive rate increases leading to unused data being retained
Solution Approach 1:
The patent combines garbage collection and data integrity checking into a single unified operation. The KV store compares block IDs in its key space against a list of in-use block IDs, simultaneously identifying both garbage blocks (present in KV store but not in use) and missing blocks (in use but not present in KV store). This merging eliminates the need for separate probabilistic data structures and their associated false positives.
2Quantity of substance
If probabilistic data structures are used for garbage collection, then storage space is reduced due to compact bit field representation, but computing resources increase due to hashing requirements
Solution Approach 1:
The patent extracts the comparison operation from the probabilistic data structure framework and performs it directly on block IDs in the KV store key space. By taking out the hashing and probabilistic matching operations, the system eliminates the computing overhead while maintaining accurate tracking of in-use blocks through direct key-space comparison.
3Ease of manufacture
If garbage collection and data integrity checking are performed as separate operations, then each operation can be optimized independently, but resource contention increases and performance degrades
Solution Approach 1:
The patent merges garbage collection and data integrity checking into a single unified operation performed concurrently by the KV store. By comparing the KV store's key space against a list of in-use block IDs, the system simultaneously identifies garbage blocks for collection and missing blocks for integrity recovery, eliminating resource contention between separate operations.
4Adaptability or versatility
If separate garbage collection and data integrity checking operations are performed, then operational flexibility is maintained, but time consumption increases
Solution Approach 1:
The patent implements continuous data integrity monitoring by integrating it into the ongoing garbage collection process. The KV store continuously compares its key space against in-use block IDs, ensuring that both garbage identification and integrity checking occur simultaneously without interruption, thereby reducing total time consumption while maintaining operational flexibility.
Data Source
AI summary
Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list may be compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.