Rolling Bloom Filter Master-Slave Deletion

Solution Overview

Problem

Bloom filters do not effectively support data set element deletion operations, leading to incorrect representation of data sets, and require significant memory resources for counting operations.

Innovation Solution

A rolling bloom filter data structure is introduced, comprising multiple sub-bloom filters that operate in a master/slave configuration, allowing for efficient insertion and deletion of elements while adhering to time-based and space-based data retention policies, reducing memory usage through a master/slave sub-data structure approach.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a counting bloom filter is used to support deletion operations, then element deletion capability is improved, but memory space requirement increases three to eight times

Engineering Contradiction:

Improveelement deletion capabilityVSAvoidmemory space requirement

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The bloom filter is divided into multiple sub-bloom filters organized in a hierarchical structure with master and slave relationships. Each sub-bloom filter handles a portion of the data set, allowing deletion operations to be performed on specific segments without affecting the entire structure. This segmentation enables the system to support deletions while using less memory than a full counting bloom filter.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a temporal dimension by organizing sub-bloom filters in generations and time windows. Elements are tracked across multiple time windows, and deletion is achieved by transitioning elements between windows rather than directly modifying bit counts. This dimensional approach enables deletion functionality without requiring the excessive memory of traditional counting bloom filters.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Quantity of substance

If a basic bloom filter is used, then memory space efficiency is improved, but element deletion operation cannot be performed

Engineering Contradiction:

Improvememory space efficiencyVSAvoidelement deletion operation

Core Design Contradiction:

Quantity of substanceVSAdaptability or versatility

Solution Approach 1:

The bloom filter is divided into multiple sub-bloom filters organized in a hierarchical structure with master and slave relationships. Each sub-bloom filter handles a portion of the data set, allowing deletion operations to be performed on specific segments without affecting the entire structure. This segmentation enables the system to support deletions while using less memory than a full counting bloom filter.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system pre-allocates multiple sub-bloom filters and establishes master/slave relationships in advance. When deletion is needed, the system has pre-configured structures ready to handle the operation, eliminating the need for complex runtime memory management that would be required in a traditional counting bloom filter.

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If data set elements are frequently inserted and deleted, then data dynamicity is improved, but data integrity may be compromised due to incorrect bit manipulation

Engineering Contradiction:

Improvedata dynamicityVSAvoiddata integrity

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The bloom filter is divided into multiple sub-bloom filters organized in a hierarchical structure with master and slave relationships. Each sub-bloom filter handles a portion of the data set, allowing deletion operations to be performed on specific segments without affecting the entire structure. This segmentation enables the system to support deletions while using less memory than a full counting bloom filter.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The master sub-bloom filter acts as an intermediary between the slave sub-bloom filters and the external system. It coordinates insertion and deletion operations across the hierarchical structure, ensuring that bit manipulations are performed correctly and data integrity is maintained during frequent dynamic operations.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS9361327B1Rolling bloom filter for data with retention policy

Publication Date: 2016.06.07 EMC IP HLDG CO LLC

AI summary

A data structure comprising two or more sub data structures representing a given data set is maintained. Each of the two or more sub data structures comprises an array of bit positions and has a set of hash functions associated therewith. Each of the hash functions is operable to map an element of the given data set to at least one of the bit positions of the array. One of the two or more sub data structures is recognized as a master sub data structure and the others of the two or more sub data structures as slave sub data structures. Insertion and deletion of elements in the data structure is based on the recognition of each of the two or more sub data structures as the master sub data structure or one of the slave sub data structures.