Indicator Text Identification Through Sample-Based Full-Text Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for analyzing unstructured documents, particularly for identifying sensitive data, are computationally expensive and inefficient, requiring the examination of all documents, which is time-consuming and resource-intensive, especially in large repositories.
Innovation Solution
A method and system that utilize a small sample of documents to analyze and determine indicator context expressions, which are then used to create queries for a search engine to identify documents containing sensitive data, reducing the computational resources and time required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all documents are examined to identify sensitive data, then identification completeness is improved, but computational cost and time consumption increase significantly
Solution Approach 1:
The patent segments the document population into a representative sample and the full population. By analyzing a carefully selected sample of documents, the system identifies sensitive data patterns and creates search queries that can be applied to the full document collection, thereby reducing the need to examine every document while maintaining identification effectiveness.
Solution Approach 2:
The patent introduces an intermediary approach by creating a sample set that serves as a mediator between the full document collection and the analysis process. The sample is used to train models and generate search queries, which then act as intermediaries to efficiently filter and identify sensitive documents in the full collection without requiring direct analysis of all documents.
2Reliability
If all documents are examined to identify sensitive data, then identification completeness is improved, but computational resources required increase significantly
Solution Approach 1:
The patent segments the computational workload by first analyzing a representative sample to identify sensitive data patterns, then using the results to create targeted search queries for the full collection. This segmentation allows the system to perform comprehensive analysis only on documents that match the search criteria, significantly reducing overall computational resource requirements.
Solution Approach 2:
The patent creates a computational copy of the analysis process by first performing detailed analysis on a sample set, then using the extracted patterns and generated queries to efficiently search the full document collection. This copying approach allows the system to maintain analysis thoroughness while reducing the computational resources needed for the complete document set.
3Measurement precision
If deeper analysis is performed on all documents, then identification accuracy is improved, but processing time and resources increase
Solution Approach 1:
The patent segments the analysis process into two stages: first, performing deep analysis on a representative sample to accurately identify sensitive data patterns; second, using the extracted patterns to create search queries that efficiently filter the full collection. This segmentation maintains high identification accuracy while improving overall processing efficiency by avoiding redundant deep analysis on all documents.
Solution Approach 2:
The patent introduces search queries as an intermediary mechanism that bridges the accurate identification achieved through sample analysis and the efficient processing of the full collection. The queries, generated from sample analysis, serve as intermediaries that enable rapid filtering and identification without requiring deep analysis of every document, thus maintaining accuracy while improving productivity.
Data Source
AI summary
Several aspects for optimizing unstructured document analysis comprise operating a document system, where the document system comprises a plurality of documents comprising unstructured content and a full-text index; receiving a request to identify documents comprising a type of data elements; selecting a sample out of the plurality of documents; determining data elements of the type in the sample of documents; determining an indicator context expression for the type of data elements out of the determined data elements of the type; determining a query for searching, using a search engine, the full-text index using the indicator context expression; and determining the documents in the document system being compliant to the determined query.