Bloom Filter Secure Search Index for Unstructured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing DLP solutions face challenges in efficiently searching unstructured documents for structured data due to large index sizes and security concerns when deploying sensitive data indexes, particularly in environments with limited memory and potential dictionary attacks.
Innovation Solution
The implementation of secure search indexes using Bloom filters, which generate hashed keys from values across multiple fields of a structured dataset, reducing memory footprint and enhancing security by distributing the search process across client and server-side devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional indexes store salted cryptographic hashes of sensitive data to maintain security, then security is improved, but the index size becomes extremely large (dozens of gigabytes or more) requiring abundant memory and dedicated servers
Solution Approach 1:
The patent extracts only the essential security property (ability to verify data presence) from traditional indexing, using Bloom filters to store probabilistic membership information instead of actual hash values. This removes the need to store large quantities of hash data while maintaining the core security function of identifying sensitive data in unstructured text.
Solution Approach 2:
The patent changes the fundamental parameter of index storage from deterministic (storing actual hashes) to probabilistic (Bloom filter bit arrays). This parameter change reduces space requirements from gigabytes to kilobytes or megabytes while accepting a controlled false positive rate, enabling deployment in environments with limited memory resources.
2Productivity
If indexes are completely stored in memory to enable efficient lookup, then search efficiency is improved, but deployment is limited to dedicated servers with abundant memory
Solution Approach 1:
The patent employs Bloom filters that can be easily instantiated and discarded in various deployment environments. These lightweight probabilistic structures can be deployed on devices with limited memory (mobile devices, edge devices, browsers) without requiring persistent large memory allocations, enabling versatile deployment across different platforms and devices.
3Ease of operation
If indexes are distributed beyond tightly controlled server environments, then accessibility is improved, but security challenges arise even with cryptographic hashing due to feasibility of dictionary attacks
Solution Approach 1:
The patent uses Bloom filters as an intermediary layer between the sensitive data and the search process. The Bloom filter provides a probabilistic membership test that prevents direct exposure of actual data values, making dictionary attacks infeasible while still enabling efficient search. This intermediary structure allows secure distribution of search functionality to client devices without compromising security.
Data Source
AI summary
The disclosed computer-implemented method for searching unstructured documents for structured data may include (1) receiving a request to search unstructured documents for a document that contains data (e.g., sensitive data) from a structured dataset, (2) generating a secure search index (e.g., a Bloom filter) for searching the unstructured documents for the sensitive data, (3) extracting a first token and a second token from an unstructured document, (4) generating a hashed key from the first token and the second token, (5) querying the secure search index to determine whether the second hashed key is contained in the secure search index, and (6) responding, upon determining that the second hashed key is contained in the secure search index, to the request with information about the unstructured document. Various other methods, systems, and computer-readable media are also disclosed.