Bloom Filter Data Structure for VM Disk Image Record Location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In replicated and decentralized storage systems for virtual machine file systems, efficiently locating specified data in a virtual machine's disk image without significant I/O resources is challenging, especially during server failures, due to the pseudo-random nature of unique id entries generated by cryptographic hash functions.
Innovation Solution
The use of Bloom filter data structures stored in memory to track the location of data records within a virtual machine disk image, allowing for efficient tracking and retrieval of data records without accessing local storage, by generating a randomized unique id and setting corresponding bits in the Bloom filter bit pattern.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data records are stored in decentralized local storage without Bloom filters, then storage autonomy and failure protection are improved, but data location efficiency deteriorates due to extensive I/O operations required to search for specified data
Solution Approach 1:
The Bloom filter data structure pre-computes and stores location information for all data records in memory before any search operations occur. When data needs to be located, the system queries the pre-built Bloom filter in O(k) time where k is the number of hash functions, eliminating the need for extensive I/O operations to search through disk-stored data. This preliminary indexing action resolves the contradiction by enabling fast data location while maintaining decentralized storage.
2Speed
If Bloom filter data structures are maintained in memory to track data locations, then data retrieval speed is improved, but memory resource consumption increases
Solution Approach 1:
The Bloom filter implementation uses parameter optimization to balance memory consumption and retrieval speed. By adjusting the number of hash functions (k), the filter size (m), and the false positive rate threshold, the system achieves efficient data location with minimized memory usage. The patent applies parameter changes to the Bloom filter configuration to optimize the trade-off between speed improvement and memory resource consumption, allowing the filter to be maintained in memory without excessive resource demands.
3Reliability
If unique ids are generated using cryptographic hash functions, then data integrity and security are improved, but data location complexity increases due to the pseudo-random nature of hash outputs
Solution Approach 1:
The Bloom filter serves as an intermediary data structure between the cryptographic hash function outputs and the physical data storage locations. Instead of directly using the pseudo-random hash outputs to locate data (which would require complex searching), the Bloom filter processes these hash values through multiple hash functions to set specific bit positions. This intermediary layer translates the secure but location-obscure hash outputs into efficient location queries, maintaining data integrity while simplifying the location process.
Data Source
AI summary
Methods and system for adding a data record to a file comprise maintaining a data structure to track locations of data records within the file, wherein each entry in the data structure is a Bloom filter that corresponds to a different portion of the file that includes a plurality of data records. When an instruction to write data to the file is received, a data record is generated to append to the file, wherein the data record comprises a randomized unique id and the data from the received instruction. A Bloom filter bit pattern is extracted from the randomized unique id and bits in a current Bloom filter entry of the data structure are set, wherein the set bits correspond to the same bit positions as the Bloom filter bit pattern. The data record is thus able to be added to the file.