Versioned Bloom Filter Bitmap Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Bloom filters face system processing delays due to the need for locking and reallocation of storage when the number of values increases, leading to increased false positives and resource wastage in searches.

Innovation Solution

Implementing a system for bloom filter versioning that allows for the creation of new bitmaps without locking the existing ones, enabling continuous read operations while writing to new bitmaps, thus preventing delays and reducing false positives by dynamically resizing bitmaps based on thresholds.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If storage allocation for bloom filter is increased to reduce false positives, then false positive rate is reduced, but system processing is delayed due to locking requirements

Engineering Contradiction:

Improvefalse positive rateVSAvoidsystem processing delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The bloom filter is divided into multiple bitmaps (first bitmap and second bitmap) that operate independently. This segmentation allows concurrent read and write operations on different bitmaps, eliminating the need for system-wide locking while maintaining low false positive rates through appropriate bitmap selection based on operation type.

Inventive Principle:

Principle #1Segmentation

2Reliability

If bloom filter storage is locked for reallocation, then storage can be properly reallocated to reduce false positives, but system processing delays occur

Engineering Contradiction:

Improvefalse positive rateVSAvoidsystem processing throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

A second bitmap is pre-generated and maintained in readiness before it is needed. When the first bitmap becomes full, the system can immediately switch to the pre-prepared second bitmap without interruption, avoiding processing delays while ensuring the bloom filter maintains optimal false positive characteristics.

Inventive Principle:

Principle #10Preliminary action

3Quantity of substance

If existing bloom filter is resized to accommodate more values, then storage efficiency is improved, but system processing is blocked during resizing

Engineering Contradiction:

Improvenumber of values trackedVSAvoidprocessing delay during resizing

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

Multiple bitmaps are merged into a single bloom filter structure that manages them collectively. The bloom filter can switch between bitmaps and manage their lifecycles (creation, activation, deletion) as a unified system, allowing seamless transitions and resource management without blocking operations.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS9886464B2Versioned bloom filter

Publication Date: 2018.02.06 SAP SE

AI summary

Disclosed herein are system, method, and computer program product embodiments for bloom filter versioning. A request to add a new value to a first bitmap is received, wherein the first bitmap includes a bit sequence corresponding to a most recently added value. It is determined that the first bitmap has exceeded a threshold. A second bitmap is generated, the second bitmap including more bits than the first bitmap. The first bitmap is deleted based upon a determination that a number of readers accessing the first bitmap is zero.