Document Classification System Using Term Frequency Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in managing and classifying a large volume of diverse inbound communications, such as emails and online forum messages, due to the inefficiency and poor results of existing clustering algorithms, which are time-consuming and often produce unrelated document clusters.
Innovation Solution
A computer-implemented method and system that uses classifiers to identify customer support issues or sentiments in documents, applying a clustering engine for grouping similar documents, and outputs the frequency of identified issues or sentiments, with a confidence threshold to ensure accurate classification and clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If clustering algorithms are used to manage a large volume of documents, then documents can be grouped into clusters, but the process is very time consuming and provides poor results with clusters having many unrelated documents
Solution Approach 1:
The patent segments the document processing task into two distinct phases: first, extracting and analyzing key terms from documents to create term-frequency profiles, and second, performing clustering based on these pre-processed term profiles. This segmentation allows the system to focus computational resources on the most relevant features, improving clustering quality while reducing overall processing time by avoiding redundant analysis of entire documents during the clustering phase.
Solution Approach 2:
The patent applies preliminary action by pre-processing documents to extract and store key term frequencies before clustering occurs. The system identifies and stores the most significant terms from each document in advance, creating a condensed representation that can be quickly compared during clustering. This preliminary term extraction step significantly reduces the time required for the actual clustering operation while maintaining or improving cluster quality.
2Measurement precision
If traditional clustering algorithms are applied to all documents, then document grouping is achieved, but the accuracy and relevance of clusters deteriorate
Solution Approach 1:
The patent applies local quality by treating different parts of the document processing system with different levels of analysis. Instead of applying uniform complex algorithms to all documents, the system extracts and emphasizes key local features (specific terms and their frequencies) that are most relevant to document similarity. This localized focus on important terms rather than uniform processing of all text improves classification accuracy while keeping the overall system manageable.
Solution Approach 2:
The patent changes parameters by transforming documents from their original full-text form into condensed term-frequency vectors with a limited number of significant terms. By changing the representation parameters from complete documents to focused term profiles with weighted frequencies, the system achieves more accurate clustering on relevant features while reducing computational complexity. The parameter transformation emphasizes discriminative terms that better distinguish between different document topics.
Data Source
AI summary
A method and system of classifying documents is provided. The method includes receiving a plurality of documents from at least one user, wherein each document includes information relating to a customer support issue or sentiment and identifying at least one customer support issue or sentiment contained within each document. The method also includes classifying the documents satisfying a confidence threshold using a classifier, clustering the remainder of the plurality of documents into groups using a clustering engine, the clustering engine applying a word analysis, and outputting a frequency of each identified customer support issue or sentiment, the frequency based on the classifying or the clustering.