Bloom Filter Mask for Database Key Identification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large datasets, identifying and maintaining referential integrity is challenging due to the degradation of primary and foreign key relationships over time, especially when systems rely on programming logic rather than data-driven mechanisms, leading to orphaned records and missing relationships.
Innovation Solution
The use of a filter mask, such as a Bloom filter, is employed to characterize the distribution of values in key fields, allowing for efficient identification of potential key relationships by generating filter keys and masks that eliminate false negatives while avoiding false positives, enabling the determination of key relationships without comparing each value directly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If direct comparison of each value in key fields is used to identify relationships, then measurement precision is improved, but productivity deteriorates due to the large volume of data requiring comparison
Solution Approach 1:
The patent introduces a filter mask as an intermediary data structure that characterizes the distribution of values in key fields. Instead of directly comparing each value between datasets, the filter mask serves as a mediator that enables efficient identification of potential key relationships by generating filter keys and masks that eliminate false negatives while avoiding false positives
Solution Approach 2:
The patent creates a filtered copy of the original data by generating a filter mask that represents the distribution of values in key fields. This filter mask is a condensed representation that can be quickly processed and compared, avoiding the need to handle the full volume of original data while preserving the essential information needed for relationship identification
2Ease of operation
If programming logic is used to maintain referential integrity, then ease of operation is improved, but reliability deteriorates due to orphaned records and missing relationships over time
Solution Approach 1:
The patent implements a feedback mechanism by using the filter mask to continuously identify and quantify referential integrity issues in datasets. The system can detect orphaned records and missing relationships by comparing filter keys against the filter mask, providing ongoing feedback about data quality that enables corrective actions to maintain reliability over time
3Productivity
If filter mask with Bloom filter is used to characterize value distribution, then productivity is improved through efficient identification, but device complexity increases due to additional data structures
Solution Approach 1:
The patent employs a Bloom filter implementation where the filter mask is generated once from the source dataset and then reused multiple times for comparing different target datasets. The filter mask acts as a disposable intermediate structure that can be quickly created and discarded, providing high processing efficiency without requiring complex persistent data structures
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for database key identification. One of the methods includes receiving an identification of a first field in a first data set, the first data set including records. The method includes identifying a set of values, the set including, for each record, a value associated with the field. The method includes generating a filter mask based on the set of values, where application of the filter mask is capable of determining that a given value is not in the set of values. The method includes receiving a second data set including a second field, the second data set including records. The method includes determining a count of a number of records in the second data set having a value associated with the second field that passes the filter mask. The method also includes storing the count in a profile.