Uniqueness Validation in Data Stores Using Hash Trees and Bloom Filters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing hash tree or Merkle-tree systems are unable to efficiently demonstrate the uniqueness of data within a set, despite their ability to verify content membership.

Innovation Solution

A method and system for probabilistic verification of uniqueness in a data store, involving the generation of an index of data elements, creation of filters, and construction of a hash tree using the index, to validate the uniqueness of data elements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If hash trees or Merkle-trees are used to verify content membership, then data integrity and content verification are improved, but the ability to demonstrate uniqueness of data within the set deteriorates

Engineering Contradiction:

Improvedata integrity verificationVSAvoiduniqueness demonstration capability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent segments the data verification process into two distinct components: a hash tree for membership verification and a Bloom filter for uniqueness verification. This segmentation allows each data structure to specialize in its respective function, with the hash tree handling content integrity and the Bloom filter handling uniqueness, thereby resolving the contradiction between verification reliability and uniqueness demonstration capability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a Bloom filter as an intermediary data structure that works in conjunction with the hash tree. The Bloom filter serves as a mediator that provides probabilistic uniqueness verification without compromising the deterministic content verification provided by the hash tree, thus enabling both functionality simultaneously

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If traditional hash trees are used for data verification, then content belonging verification is achieved, but probabilistic verification of uniqueness deteriorates

Engineering Contradiction:

Improvecontent belonging verification accuracyVSAvoiduniqueness verification reliability

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent merges two different data structures - the deterministic hash tree and the probabilistic Bloom filter - into a unified verification system. The hash tree provides precise content belonging verification while the Bloom filter adds probabilistic uniqueness verification, and both work together to provide comprehensive data validation that addresses both measurement precision and reliability requirements

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS12206781B2Method and system to validate uniqueness in a data store

Publication Date: 2025.01.21 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method, system and apparatus of for validating uniqueness of a non-disclosed element of data, including in response to receiving a plurality of elements, generating an index of the plurality of elements of data received from a data store to create uniquely indexed data, generating filters from the index, generating a hash tree for the uniquely indexed data using the index, and validating a unique element of data from the hash tree and filters.