Bloom Filter Lookup Tables for Memory-Efficient Multi-Token Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for identifying specific data types in digital documents require significant computing resources and time, leading to inefficiencies and increased security risks due to prolonged processing times, especially when handling large volumes of data.
Innovation Solution
A digital document search system utilizing a probabilistic data structure, such as a Bloom filter, to represent lookup lists, which reduces memory usage and limits token searches by iteratively reducing the number of tokens compared, thereby optimizing memory and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional systems use tries to detect data types in digital documents, then detection accuracy is maintained, but memory usage increases significantly
Solution Approach 1:
The patent uses Bloom filters as a probabilistic data structure that creates a compressed representation (copy) of the lookup list. Instead of storing the complete lookup list in memory, the system hashes lookup list items and stores only the hash values in the Bloom filter, significantly reducing memory requirements while maintaining detection capability.
Solution Approach 2:
The patent changes the data structure from a traditional trie (prefix tree) to a Bloom filter, which transforms the storage approach from storing complete token sequences to storing hashed parameter values. This parameter transformation enables memory-efficient storage while maintaining the ability to detect data types through probabilistic matching.
2Reliability
If conventional systems compare large volumes of documents to large lookup lists, then detection completeness is improved, but processing time increases significantly
Solution Approach 1:
The patent pre-processes the lookup list by hashing all items and storing the hashes in the Bloom filter before actual document processing. This preliminary action enables fast lookup during document scanning, as the system only needs to compute hashes for tokens being processed rather than performing complex pattern matching against large lookup lists during the actual detection phase.
Solution Approach 2:
The patent replaces the mechanical process of comparing document tokens against entire lookup list entries with a probabilistic hashing mechanism. Instead of performing string matching and pattern comparison, the system uses hash function transformations to quickly determine potential matches, significantly reducing processing time while maintaining detection completeness.
3Reliability
If conventional systems process large amounts of data over large amounts of time, then processing thoroughness is improved, but security risks increase due to prolonged exposure
Solution Approach 1:
The patent replaces slow mechanical pattern matching with fast probabilistic hashing operations. By using Bloom filters with pre-computed hash values, the system can thoroughly process large datasets much faster, reducing the time window for security risks while maintaining processing thoroughness through comprehensive token scanning.
4Reliability
If conventional systems use lookup lists with gigabytes of data, then detection accuracy is maintained, but computing resources required increase significantly
Solution Approach 1:
The patent creates a compressed copy of the lookup list in the form of a Bloom filter, storing only the essential hash information rather than the complete data. This copying approach maintains detection accuracy for identifying data types while reducing computing resource requirements for memory allocation and data processing.
Solution Approach 2:
The patent transforms the lookup list from a data-intensive structure to a parameter-efficient structure by using hashing. The system changes from storing and processing full token sequences to storing and processing hash parameter values, which requires significantly fewer computing resources while maintaining detection capability.
Data Source
AI summary
Methods, systems, and non-transitory computer readable storage media are disclosed for optimizing computer memory usage for lookup lists in computer memory via probabilistic data structures. For example, the disclosed system generates a probabilistic data structure (e.g., a Bloom filter) to represent data in a lookup list including multi-token items by hashing items of the lookup list to sets of bit values in a bit vector. The disclosed system classifies text content in a digital document by utilizing a maximum number of tokens from multi-token items in the lookup list to select and compare sets of sequential tokens in the digital document to the probabilistic data structure. The disclosed system also iteratively reduces the number of tokens in sets of sequential tokens for subsequent comparisons. Furthermore, in some aspects, the disclosed system causes a computing device to modify a digital document and/or database operations based on the classifications.