Bloom Filter Lookup Structures for Memory-Efficient Multi-Token Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for identifying specific data types in digital documents using lookup lists are inefficient in terms of memory usage and processing time, particularly when handling large volumes of data, which can lead to security risks and compliance issues due to the high computational resources required.
Innovation Solution
The use of probabilistic data structures, such as Bloom filters, to represent lookup lists, which reduces memory usage and processing time by hashing items into bit vectors and iteratively reducing the number of tokens in sequential comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional trie structures are used to detect data types in digital documents, then the system can detect matches with the longest possible sequence, but the system requires a large amount of computer memory and CPU resources
Solution Approach 1:
The patent creates a compressed representation (copy) of the trie structure by hashing nodes to bit vectors. Instead of storing the complete trie in memory, it generates a condensed version that preserves the essential matching capabilities while occupying significantly less memory space, thus resolving the contradiction between detection accuracy and memory consumption
Solution Approach 2:
The patent transforms the trie structure from a node-based hierarchical representation to a bit vector-based representation through hashing. This parameter change in the data structure's fundamental organization allows the system to maintain match detection functionality while dramatically reducing memory requirements by eliminating the overhead of node objects and pointers
2Measurement precision
If conventional trie structures are used to detect data types in digital documents, then the system can detect matches with the longest possible sequence, but the system requires significant CPU resources for comparison
Solution Approach 1:
The patent replaces the mechanical traversal process of conventional tries (which requires navigating through multiple node levels and performing sequential comparisons) with a hashing-based direct access mechanism. This substitution eliminates the need for iterative node traversal and reduces CPU-intensive operations to simple hash computations and bit vector comparisons, thereby significantly lowering power and computational resource requirements
3Measurement precision
If direct comparisons of text content to large lookup lists are performed, then the system can identify specific data types accurately, but the processing time increases significantly
Solution Approach 1:
The patent performs preliminary hashing of the lookup list items to generate bit vectors before the actual scanning process. This pre-processing step organizes the search data into an optimized structure that enables faster querying during the scanning phase, thereby reducing the processing time required for accurate data type identification without sacrificing detection accuracy
4Measurement precision
If conventional systems process large amounts of data over large amounts of time, then they can detect data types, but higher-priority data is exposed to security risks
Solution Approach 1:
The patent implements preliminary anti-action by using the hashed bit vector structure to enable faster detection and processing of sensitive data types. By reducing the time required to scan and identify data types through efficient hashing and bit vector comparisons, the system minimizes the window of exposure during which data remains unprocessed and vulnerable to security risks such as breaches or unauthorized access
Data Source
AI summary
Methods, systems, and non-transitory computer readable storage media are disclosed for optimizing computer memory usage for lookup lists in computer memory via probabilistic data structures. For example, the disclosed system generates a probabilistic data structure (e.g., a Bloom filter) to represent data in a lookup list including multi-token items by hashing items of the lookup list to sets of bit values in a bit vector. The disclosed system classifies text content in a digital document by utilizing a maximum number of tokens from multi-token items in the lookup list to select and compare sets of sequential tokens in the digital document to the probabilistic data structure. The disclosed system also iteratively reduces the number of tokens in sets of sequential tokens for subsequent comparisons. Furthermore, in some aspects, the disclosed system causes a computing device to modify a digital document and/or database operations based on the classifications.