Probabilistic Data Structures for Database Entry Length Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database search methods, particularly regular expression matching, are inefficient and prone to high false positives when searching large databases for specific information like personally identifying information, such as UUIDs, leading to slow performance and inaccurate results.
Innovation Solution
A computing device sorts database entries into length sets and generates probabilistic data structures, like Bloom filters, to quickly determine the presence of input strings within predefined length ranges, reducing false positives and improving search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If regular expression matching is used to search database entries, then the search can identify entries matching a specified pattern, but the search speed becomes prohibitively slow when searching very large databases
Solution Approach 1:
The patent segments the database into multiple buckets based on the length of database entries. Each bucket contains entries of a specific length, allowing the system to search only relevant buckets rather than the entire database. This segmentation enables faster search by reducing the search space while maintaining accurate pattern matching through probabilistic data structures within each bucket.
2Productivity
If regular expression matching is used to search for personally identifying information, then the search can identify entries matching the pattern, but the false positive rate becomes extremely high
Solution Approach 1:
The patent changes the search parameter from pattern matching alone to a two-stage approach: first filtering by entry length (a discrete parameter), then using probabilistic data structures for further identification. This parameter change reduces false positives by adding a length validation layer that distinguishes between entries that merely match the pattern and those that are actually personally identifying information.
3Quantity of substance
If the database size is increased to handle more data, then the coverage of the search increases, but the search time becomes prohibitively slow
Solution Approach 1:
The patent segments the large database into smaller, manageable buckets organized by entry length. This segmentation allows the system to scale with database size while maintaining fast search performance, as the search algorithm only needs to examine relevant length buckets rather than scan the entire database sequentially.
Solution Approach 2:
The patent performs preliminary organization of data by length during database loading, creating pre-sorted buckets before search operations. This preliminary action enables fast search by eliminating the need to scan the entire database during search operations, as the data is already organized in a search-friendly structure.
Data Source
AI summary
According to one aspect of the present disclosure, a computing device is provided, including non-volatile memory storing a database including a plurality of database entries. The computing device may further include a processor configured to sort the plurality of database entries into a plurality of database entry length sets. For each database entry length set, each database entry included in the database entry length set may be within a predefined length range. For each database entry length set, the processor may be further configured to generate a probabilistic data structure based on the one or more database entries included in the database entry length set.