Semi-Supervised Document Classification for Confidentiality Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data loss prevention solutions struggle to accurately identify confidential information in electronic documents due to high false positive rates and the need for manual annotation, which is time-consuming and prone to human error, while existing context-based methods fail to account for document variability across different categories.
Innovation Solution
A multi-staged and multi-pipelined process using semi-supervised and supervised learning to train classifiers for each document category, utilizing positive, negative, and unlabeled samples to determine the probability of a document containing confidential information, with pipelines tailored to handle varying sample distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation methods are used to tag electronic documents to identify confidential or sensitive information, then the accuracy of identification can be improved, but the time consumption and cost increase significantly
Solution Approach 1:
The system performs preliminary automated classification of documents using machine learning models before final review. This preliminary action processes the bulk of documents automatically, reserving manual annotation only for cases where automated classification is uncertain or confidence is low, thereby significantly reducing overall time consumption while maintaining high identification accuracy
Solution Approach 2:
The patent introduces an automated machine learning-based classification system as an intermediary between raw documents and final confidential information identification. This intermediary layer processes documents through multiple pipelines and models, providing pre-filtered results that reduce the burden on manual annotation while preserving accuracy for truly sensitive documents
2Measurement precision
If manual annotation methods are used to tag electronic documents, then the accuracy of identification can be improved, but the cost becomes prohibitive
Solution Approach 1:
The system enables self-service automated classification where the machine learning model independently processes and classifies documents without requiring extensive manual intervention. The model serves itself by continuously learning from labeled data and improving its own performance, reducing both time and cost while maintaining high identification accuracy
Solution Approach 2:
The patent uses synthetic or augmented data copies to train and validate the classification model, reducing the need for extensive manual annotation of real confidential documents. By working with copied or synthesized training data, the system achieves high accuracy without the prohibitive costs of manually tagging every actual confidential document
3Measurement precision
If context-based methods are used for document classification, then the accuracy for specific contexts can be improved, but the system fails to account for document variability across different categories
Solution Approach 1:
The patent implements a universal classification framework that handles multiple document categories and contexts through a single multi-pipeline system. The framework is designed to be category-agnostic, adapting to different document types (legal, medical, financial, etc.) without requiring separate specialized systems, thereby achieving both contextual accuracy and broad category adaptability
Solution Approach 2:
The system applies local quality by allowing different classification pipelines and models to be optimized for specific document categories or contexts while operating under a unified framework. Each pipeline can be tailored to the specific characteristics of certain document types, providing high contextual accuracy for each category while maintaining overall system versatility
4Ease of operation
If existing data loss prevention solutions are used, then the implementation can be simplified, but the false positive rate remains high
Solution Approach 1:
The system incorporates feedback mechanisms where classification results are continuously evaluated and used to retrain and improve the models. False positives are identified through feedback loops and used to refine the classification criteria, progressively reducing the false positive rate while maintaining implementation simplicity through automated model updates
Solution Approach 2:
The patent implements dynamic classification thresholds and model parameters that adapt based on the specific document being classified and the confidence level of the model. Rather than using fixed simple rules, the system dynamically adjusts its decision criteria to minimize false positives while maintaining ease of operation through automated parameter optimization
Data Source
AI summary
A method, a computing device, and a non-transitory machine-readable medium for classifying documents. A document collection is sorted into a plurality of categories. A classifier corresponding to a category of the plurality of categories is trained to output a probability that a document associated with the category is of a selected type (e.g., confidential). The training includes determining, by the processor, that a cardinality of a set of negative samples in a train set is not above a pipeline threshold but is at least one and training the classifier via a first pipeline and a second pipeline using a training group that includes a first portion of a group of positive samples in the train set, a second portion of a set of negative samples in the train set, and a third portion of a group of unlabeled samples in the train set.