Segmented Bloom Filters for Distributed Node Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Bloom filters used in distributed systems for synchronizing states between computing nodes are prone to false positives, leading to inconsistencies and vulnerabilities, especially as the number of nodes increases, which necessitates larger filters that compromise efficiency and responsiveness.
Innovation Solution
A system where each pair of computing nodes generates and transmits unique Bloom filters based on hash values of data entities, determining consistency without sending whole data sets, reducing false positives and improving memory efficiency, and enabling verified recordal of data entities across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the size of the Bloom filter is increased to reduce false positive responses, then system reliability is improved, but data memory utilization efficiency deteriorates and synchronization time increases
Solution Approach 1:
The system divides the Bloom filter into multiple segments, each handling a portion of the data entities. This segmentation allows the filter to maintain high reliability through comprehensive coverage while keeping each segment compact for efficient memory usage. The segmented structure enables parallel processing and reduces the memory footprint compared to a single large filter.
Solution Approach 2:
The patent introduces a hierarchical dimension to the Bloom filter structure, organizing filters at multiple levels. This dimensional approach allows the system to achieve high reliability through layered verification while maintaining efficiency at each level, avoiding the need for a single large-scale filter that would consume excessive memory.
2Reliability
If the size of the Bloom filter is increased to reduce false positive responses, then system reliability is improved, but system responsiveness deteriorates
Solution Approach 1:
By segmenting the Bloom filter into smaller units, the system can process and verify data entities in parallel across segments. This segmentation reduces the time required to traverse the entire filter structure, maintaining high system responsiveness while achieving reliable verification through comprehensive segment coverage.
Solution Approach 2:
The system performs preliminary filtering using smaller, faster Bloom filter segments before conducting more thorough verification. This preliminary action eliminates obvious mismatches quickly, reducing the overall time required for state synchronization while maintaining reliability through subsequent verification steps.
3Quantity of substance
If conventional Bloom filters are used for state synchronization, then data memory efficiency is improved, but false positive responses increase leading to inconsistencies
Solution Approach 1:
The system implements feedback mechanisms where verification results from Bloom filter segments are used to guide subsequent verification steps. When false positives are detected, the feedback loop triggers additional verification processes, ensuring data consistency while maintaining the memory efficiency of the Bloom filter structure.
Solution Approach 2:
The patent introduces intermediary verification structures between the Bloom filter and the final state synchronization decision. These intermediaries act as mediators that resolve false positives by providing an additional layer of verification, ensuring data consistency without requiring the Bloom filter itself to be enlarged.
Data Source
AI summary
There is disclosed a system (100) comprising computing nodes (102A, 102B, 102C, 102D), wherein each computing node includes processor (104A, 104B, 104C, 104D), wherein each computing node comprises hash value of each data entity of pre-existing data entities therein, and wherein system (100) operates to provide verified recordal of data entities therein to ensure consistent recordal of data that assists to ensure reliable, efficient and robust operation of the system (100). The processor (104A, 104B, 104C, 104D) is configured to perform process comprising: executing first set of instructions on first computing node (102A) and second computing node (102B) to generate first set of values; executing second set of instructions on the hash value of each pre-existing data entity of first computing node (102A) and first set of values, to generate second set of values; generating data structure based on the second set of values; transmitting data structure; determining a mapping factor based on pre-existing data entities in second computing node (102B); and transmitting data entities from first computing node (102A) to second computing node (102B) if the mapping factor is less than 100 percent.