Document Classification via Query Filtering and Supervised Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated document classification techniques, such as supervised classifiers and search-engine-based methods, face inefficiencies in processing large volumes of documents, with supervised classifiers being slow and search-engine-based approaches lacking accuracy.
Innovation Solution
A system that uses a supervised classifier to generate a representative feature set and provides parameters or output decisions to a query generator, which formulates an index query to filter documents, reducing the number processed by the classifier, utilizing an existing document index for faster processing and improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised classifiers are used to classify documents, then classification accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments the document classification process into two distinct stages: (1) a fast filtering stage using search-engine queries to identify candidate documents, and (2) a precise classification stage using supervised classifiers only on the filtered candidates. This segmentation allows the system to maintain high accuracy while significantly improving processing speed by avoiding exhaustive classification of all documents.
Solution Approach 2:
The patent introduces an intermediary component - the search-engine query system - that acts as a mediator between the document set and the supervised classifier. This intermediary filters out irrelevant documents before they reach the classifier, reducing the classifier's workload while preserving accuracy for relevant documents.
2Speed
If search-engine queries are used to classify documents, then processing speed is improved, but classification accuracy deteriorates
Solution Approach 1:
The classification process is segmented into a rough filtering phase (search-engine queries) and a precise classification phase (supervised classifiers). The search-engine queries provide a fast but imprecise initial filter, while the supervised classifiers provide precise classification for the filtered candidates, achieving both speed and accuracy.
Solution Approach 2:
The system performs partial classification using search-engine queries on the entire document set, then applies exhaustive supervised classification only to the subset of candidate documents. This partial action approach achieves speed benefits without sacrificing overall accuracy.
3Reliability
If all documents are processed by the supervised classifier, then classification completeness is improved, but processing time increases
Solution Approach 1:
The patent extracts and processes only the relevant subset of documents that are likely to belong to the target category, using search-engine queries to identify candidates. This extraction approach maintains classification completeness for relevant documents while eliminating waste of time on irrelevant documents.
Solution Approach 2:
The system performs preliminary filtering using search-engine queries before applying the supervised classifier. This preliminary action identifies candidate documents in advance, ensuring that no relevant documents are missed while reducing the overall processing time by excluding irrelevant documents from classification.
Data Source
AI summary
Provided are, among other things, systems, methods and techniques for classifying a collection of documents. A term is identified based on an indication of ability of the term's presence within a given document to predict whether the given document should be classified into an identified category. A document index is then queried using the identified term and, in response, search results that define a candidate set of documents are received. Finally, a classifier is applied to documents within the candidate set to determine which of the documents should be classified into the identified category.