Text Classification System Using Shannon Information Theory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual classification of large numbers of text documents for analysis is not feasible, necessitating the development of computerized methods that can be trained on a small set of classified messages and applied to a large set of unclassified ones, while maintaining high recall and precision.
Innovation Solution
A system comprising a trainer, classifier, and search engine that extracts features from documents, generates a classification model, and indexes results, using Shannon Information Theory to select significant features and Bayes law for probability calculations, allowing for efficient classification and indexing of text communications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual classification methods are used, then classification accuracy can be maintained, but productivity drops to near zero for large volumes of text
Solution Approach 1:
The system performs preliminary action by training the classification model in advance using a small set of manually classified messages. This pre-training phase captures sentiment patterns and linguistic features, enabling the model to automatically classify large volumes of new messages without requiring manual intervention for each message, thus resolving the contradiction between productivity and accuracy
Solution Approach 2:
The system creates a computational model that copies the classification behavior of human experts. By training on manually classified examples, the model learns to replicate human sentiment analysis capabilities, allowing automated classification that maintains accuracy while dramatically increasing productivity
2Productivity
If computerized classification is implemented, then productivity increases, but recall and precision may deteriorate without proper training
Solution Approach 1:
The system performs preliminary training using a small set of manually classified messages before deploying automated classification. This pre-training phase is crucial for establishing reliable classification boundaries and sentiment thresholds, ensuring that the automated system maintains high recall and precision while processing large volumes of messages
Solution Approach 2:
The system incorporates feedback mechanisms where classification results can be reviewed and corrected. Misclassified messages provide feedback that can be used to refine the model, improving recall and precision over time while maintaining high productivity through automated processing
Data Source
AI summary
Methods and apparatus to classify text communications are disclosed. An example method includes determining a first score indicating a likelihood that a text belongs to a first classification mode by combining a first sentence score and a second sentence score retrieved from an index, the first sentence score indicating a probability that a first sentence in the text belongs to the first classification mode, the second sentence score indicating that a second sentence following the first sentence belongs to the first classification mode, determining a second score indicating a likelihood that the text belongs to a second classification mode, comparing the first score to the second score, classifying the text as the first classification mode when the first score is greater than the second score, and determining a confidence level that the text belongs to the first classification mode by dividing the first score by the second score.