Spam Filtering Feature Selection via Entropy Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing spam filtering technologies face challenges in efficiently selecting and maintaining effective features to differentiate between spam and legitimate emails, leading to high false positive rates and increased computational complexity, especially in enterprise-class systems.
Innovation Solution
The system calculates entropy values for features and removes those that increase overall entropy, while retaining features that decrease entropy, and evaluates time-consuming features based on information gain and timing costs to optimize feature sets and reduce false positives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If more features are added to the spam filtering model to improve detection accuracy, then the ability to differentiate between spam and ham improves, but the computational complexity and processing time increase
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the entropy threshold parameter based on training data characteristics. The system calculates entropy for each feature and compares it against an adaptive threshold, allowing the filtering mechanism to optimize its sensitivity without manually adjusting the entire feature set. This resolves the contradiction by enabling accurate spam detection through parameter optimization rather than feature proliferation.
Solution Approach 2:
The patent implements preliminary action by pre-calculating entropy values for all features during the training phase and storing them for rapid retrieval during message filtering. This pre-processing step eliminates the need to recalculate entropy during real-time filtering, thereby maintaining high detection accuracy while significantly reducing computational complexity and processing time during operation.
2Measurement precision
If more features are added to the spam filtering model to improve detection accuracy, then the ability to differentiate between spam and ham improves, but the processing time increases
Solution Approach 1:
The patent implements preliminary action by pre-calculating entropy values for all features during the training phase and storing them for rapid retrieval during message filtering. This pre-processing step eliminates the need to recalculate entropy during real-time filtering, thereby maintaining high detection accuracy while significantly reducing computational complexity and processing time during operation.
3Reliability
If the feature set is expanded to reduce false positives, then the reliability of spam filtering improves, but the model becomes more prone to over-fitting
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the entropy threshold parameter based on training data characteristics. The system calculates entropy for each feature and compares it against an adaptive threshold, allowing the filtering mechanism to optimize its sensitivity without manually adjusting the entire feature set. This resolves the contradiction by enabling accurate spam detection through parameter optimization rather than feature proliferation.
4Measurement precision
If computational resources are increased to handle more features, then the detection accuracy improves, but the cost and resource consumption increase
Solution Approach 1:
The patent implements preliminary action by pre-calculating entropy values for all features during the training phase and storing them for rapid retrieval during message filtering. This pre-processing step eliminates the need to recalculate entropy during real-time filtering, thereby maintaining high detection accuracy while significantly reducing computational complexity and processing time during operation.
Data Source
AI summary
A system and method for removing ineffective features from a spam feature set. In particular, in one embodiment of the invention, the an entropy value is calculated for the feature set based on the effectiveness of the feature set at differentiating between ham and spam. Features are then removed one at a time and the entropy is recalculated. Features which increase the overall entropy are removed and features which decrease the overall entropy are retained. In another embodiment of the invention, the value of certain type of time consuming features (e.g., rules) is determined based on both the information gain associated with the features and the time consumed implementing the features. Those features which have relatively low information gain and which consume a significant amount of time to implement are removed from the feature set.