Text Categorization Using Multi-Clustering Probability Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text categorization methods face low accuracy due to limitations in clustering algorithms, and the use of neural networks for categorization models leads to lengthy training times and inaccurate sample selection, affecting overall categorization precision.
Innovation Solution
A two-stage approach involving preliminary clustering followed by accurate categorization, using a preset clustering method to perform initial clustering and then determining a statistical result based on clustering probabilities to improve categorization accuracy without requiring neural network training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a preset clustering method is used for text categorization, then the processing speed is fast, but the clustering accuracy is limited resulting in low categorization accuracy
Solution Approach 1:
The patent performs preliminary clustering using a preset clustering method to obtain initial clustering results, which serve as the foundation for subsequent probability calculation and refined categorization. This preliminary action enables the system to quickly group texts before applying more computationally intensive probability-based refinement.
Solution Approach 2:
The patent introduces clustering probability as an intermediary metric between the initial clustering results and final categorization. By calculating the probability that texts belong to the same cluster based on multiple clustering operations, the system creates a intermediate representation that bridges the gap between fast but inaccurate clustering and accurate but slow categorization.
2Measurement precision
If multiple clustering operations are performed to improve accuracy, then the categorization accuracy improves, but the processing time increases
Solution Approach 1:
The patent performs multiple clustering operations (excessive action) to improve accuracy, but only uses the statistical results from these operations to calculate clustering probabilities rather than performing full re-categorization. This partial use of multiple clustering results achieves improved accuracy while limiting the time cost.
Solution Approach 2:
The patent changes the parameter from direct clustering results to clustering probability derived from multiple clustering operations. By transforming the output parameter and using statistical aggregation, the system achieves better accuracy through multiple operations while the probability calculation provides a efficient summary metric.
3Measurement precision
If neural networks are used for text categorization, then the categorization accuracy can be improved, but the training time becomes lengthy and sample selection accuracy is affected
Solution Approach 1:
The patent replaces expensive and time-consuming neural network training with a cheaper alternative based on preset clustering methods and probability calculations. The clustering-based approach serves as a disposable, lightweight solution that achieves comparable accuracy without the lengthy training process and sample selection issues of neural networks.
Solution Approach 2:
The patent substitutes the complex neural network mechanical system with a simpler statistical mechanics approach based on clustering operations and probability calculations. This replacement eliminates the need for iterative training while maintaining categorization accuracy through mathematical probability analysis.
Data Source
AI summary
An embodiment of the present application provides a text categorization method and apparatus, after obtaining a first text set, performing multiple clustering on the texts in the plurality of text pairs, and obtaining a second text set in each clustering; and for each text pair, calculating a clustering probability that all the texts in the text pair are clustered into the same second text set in the multiple clustering; and then categorizing the first text set into a plurality of third text sets based on the clustering probability of each text pair; in this way, a preliminary clustering is performed on the to-be-categorized texts, and then an accurate categorization is performed in combination with a preliminary clustering result to obtain a final text categorization result, not directly taking the text clustering result as the final text categorization result, which can improve the accuracy of text categorization.