Short Text Classification via Improved Chi-Square and TF-IWF Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current short text classification algorithms face challenges due to sparse semantic features, high dimensionality, and low classification accuracy, particularly when dealing with large volumes of short text data from social networks, where traditional methods fail to effectively select features and assign appropriate weights.
Innovation Solution
A text classification system based on feature selection using improved chi-square statistics and TF-IWF weighting, which introduces word frequency adjustment parameters, intra-category position parameters, and negative correlation correction factors to enhance feature subset selection and assign weights to feature entries, thereby improving classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional classification algorithms are directly applied to short texts, then the processing speed is fast, but the classification accuracy is low due to sparse semantic features
Solution Approach 1:
The patent segments the feature selection process into multiple stages: initial feature extraction from short texts, filtering based on statistical criteria (chi-square, information gain), and refined selection using weighting schemes. This multi-stage segmentation allows the system to handle the complexity of feature selection systematically while improving classification accuracy from the sparse features inherent in short texts.
Solution Approach 2:
The patent transforms the feature selection problem from a single-dimension approach to a multi-dimensional framework by introducing multiple statistical criteria (chi-square statistic, information gain, term frequency-inverse document frequency) and combining them through weighting mechanisms. This dimensional expansion allows the system to evaluate features from multiple perspectives simultaneously, resolving the contradiction between accuracy and complexity.
2Productivity
If feature dimensionality is reduced through feature selection, then the classification efficiency is improved, but the loss of semantic information occurs
Solution Approach 1:
The patent changes the parameters of feature evaluation by introducing multiple statistical metrics (chi-square values, information gain scores, TF-IDF weights) and dynamically adjusting their importance through weighting schemes. This parameter transformation allows the system to identify and retain the most semantically important features while reducing dimensionality, thereby maintaining classification efficiency without significant semantic information loss.
Solution Approach 2:
The patent implements feedback mechanisms where the classification performance is continuously evaluated and used to adjust the feature selection criteria. The system learns from classification results and refines its feature selection process, ensuring that the reduced feature set maintains sufficient semantic information for accurate classification while achieving improved efficiency.
3Measurement precision
If multiple feature selection criteria are introduced to improve accuracy, then the classification performance is enhanced, but the computational complexity increases
Solution Approach 1:
The patent applies partial action by selectively applying different feature selection criteria to different subsets of features or different text categories. Rather than computing all possible statistical measures for all features uniformly, the system strategically applies criteria where they provide the most benefit, reducing overall computational resource consumption while maintaining classification accuracy.
Data Source
AI summary
The present disclosure discloses a text classification system based on feature selection and a method thereof in the technical field of natural language processing and short text classification, comprising: acquiring a text classification data set; dividing the text classification data set into a training text set and a test text set, and then pre-processing the training text set and the test text set; extracting feature entries from the pre-processed training text set through improved chi-square statistics to form feature subsets; using TF-IWF algorithm to give the weight to the extracted feature entries; based on the weighted feature entries, establishing a short text classification model based on a support vector machine; and classifying the pre-processed test text set by the short text classification model. The present disclosure solves the problem that the short text content is sparse to some extent, thereby improving the performance of short text classification.