Bit Array Incremental Scanning for Sensitive Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Data Loss Prevention (DLP) solutions are inefficient as they require re-scanning all documents every time, even if they have already been scanned, leading to unnecessary resource utilization and inefficiency in identifying newly introduced or modified documents containing sensitive information.
Innovation Solution
The implementation of incremental scanning using bit arrays, where documents are associated with a bit array to track their scanning status, allowing the system to refrain from re-scanning documents that have already been scanned, and only scan those that have been modified or introduced since the last scan, utilizing hash functions to generate keys and identify positions in the bit array for efficient scanning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all documents are re-scanned every time, then sensitivity in detecting sensitive information is improved, but productivity deteriorates due to unnecessary re-scanning of already scanned documents
Solution Approach 1:
The system performs preliminary actions by maintaining a bit array that pre-records which documents have been scanned. Before scanning, the system checks the bit array to identify documents that need scanning, thus avoiding unnecessary re-scanning of already processed documents while maintaining sensitivity to detect sensitive information in new or modified documents.
Solution Approach 2:
The system uses feedback mechanisms by continuously updating the bit array with scanning status information. The bit array provides feedback about which documents have been scanned, enabling the system to make informed decisions about which documents require scanning in the next iteration, thereby improving productivity without compromising sensitivity.
2Measurement precision
If a database is used to track scanned documents, then accuracy in tracking is improved, but device complexity worsens due to database management overhead
Solution Approach 1:
The system replaces complex database structures with simple bit arrays that use minimal storage space (one bit per document). This lightweight data structure provides sufficient tracking accuracy while dramatically reducing system complexity, memory usage, and computational overhead compared to traditional database solutions.
Solution Approach 2:
The system changes the parameter of data structure representation from complex database records to simple bit arrays. Each document's scanning status is represented by a single bit value (0 or 1), transforming the tracking mechanism into a highly efficient, low-complexity system that maintains accurate tracking information.
3Productivity
If bit arrays are used to track scanned documents, then productivity is improved by reducing re-scanning, but loss of information worsens due to potential false positives in determining scan status
Solution Approach 1:
The system performs preliminary checking against the bit array before scanning documents. By pre-identifying documents that have been scanned through the bit array, the system avoids unnecessary re-scanning while ensuring that documents requiring scanning are properly processed, thus maintaining scan status accuracy while improving productivity.
Solution Approach 2:
The bit array provides continuous feedback about document scanning status. After each scanning operation, the system updates the bit array to reflect the current state, ensuring that the information remains accurate and up-to-date. This feedback mechanism prevents loss of information by maintaining reliable tracking of which documents have been scanned.
Data Source
AI summary
A method for incremental scanning of documents may include identifying documents to be scanned for sensitive information according to at least one data loss prevention (DLP) policy, the documents being associated with a bit array. The method may further include generating a key of a documents and applying, for each document, a set of functions to the key of the document, where outputs of the functions map the key to positions in the bit array. The method further includes identifying a subset of the positions in the bit array that correspond to the outputs of the functions in the set of functions, the subset of the positions to indicate whether the document has been scanned.