Document Classification via Frequency Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for classifying large volumes of business documents are inefficient, as keyword searches lack context discrimination and manual tagging is impractical for extensive document collections, especially when incorporating external content with varied structures and formats.
Innovation Solution
A method and system for automatically classifying documents based on a predefined hierarchy, involving the generation of aliases and search strategies, where frequency analysis and safety scores are used to determine document classification within the hierarchy, allowing for context-based classification without relying on pre-existing tags.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If keyword search is used for document classification, then search speed is improved, but classification accuracy deteriorates due to lack of context discrimination
Solution Approach 1:
The patent segments the document classification process into multiple stages: generating multiple variants for each hierarchy object, calculating frequency thresholds for each variant, and evaluating documents against these variants. This segmentation allows the system to maintain search speed while improving classification accuracy through context-aware frequency analysis.
Solution Approach 2:
The patent changes the parameter of classification from simple keyword matching to frequency-based variant matching. By calculating frequency thresholds and comparing document object frequencies against these thresholds, the system transforms the classification mechanism to achieve both speed and accuracy.
2Measurement precision
If manual tagging is used for document classification, then classification accuracy is improved, but productivity deteriorates due to impracticality for extensive document collections
Solution Approach 1:
The patent implements self-service classification where the system automatically generates variants, calculates frequency thresholds, and classifies documents without human intervention. The frequency analysis mechanism enables the system to serve itself, achieving both accuracy and high productivity for extensive document collections.
Solution Approach 2:
The patent replaces the mechanical manual tagging process with an automated frequency analysis system. By substituting human manual classification with algorithmic frequency threshold comparison, the system maintains accuracy while dramatically improving productivity.
3Productivity
If frequency threshold is set low for better recall, then more documents are classified, but measurement precision deteriorates due to false positives
Solution Approach 1:
The patent makes the frequency threshold dynamic rather than static. By calculating frequency thresholds based on actual document collection statistics and variant occurrence frequencies, the system dynamically adjusts thresholds to maintain precision while maximizing classification volume. This dynamic approach prevents both false positives and false negatives.
Data Source
AI summary
A method of classifying individual documents in a document collection according to a hierarchy may include selecting an object from the hierarchy, generating one or more variants for the object, and for each of the one or more variants, determining a frequency threshold based at least in part on how frequently the one or more variants occurs in the document collection. The method may also include selecting a first document in the document collection, where the first document includes one or more objects that match at least one of the one or more variants. The method may additionally include determining that the number of the one or more objects exceeds the frequency threshold and classifying the first document with the object in the hierarchy.