Hybrid Duplicate Detection and Inductive Classifier for Spam Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The proliferation of spam emails poses a significant challenge for both email users and providers, as they consume resources and are disruptive, with existing technologies struggling to effectively filter out near-identical yet varied spam messages.
Innovation Solution
A hybrid system combining a signature-based duplicate detector and an inductive classifier, where the inductive classifier generates attribute information used to create a lexicon, allowing the duplicate detector to identify near-duplicate spam emails and classify unknown documents based on class signatures and query signatures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a duplicate detector uses a lexicon of attributes to identify near-duplicate spam emails, then spam filtering effectiveness is improved, but the complexity of lexicon creation increases
Solution Approach 1:
The inductive classifier performs preliminary analysis of training documents to generate attribute information and select important attributes before the duplicate detector uses them to create the lexicon. This preliminary action simplifies the lexicon creation process by providing pre-processed, relevant attribute data.
Solution Approach 2:
The attribute information generated by the inductive classifier serves multiple purposes: it is used both for classification model development and for lexicon creation in duplicate detection. This multi-functionality reduces redundancy and simplifies the overall system by sharing the same attribute analysis between two different components.
2Measurement precision
If the inductive classifier analyzes all attributes in training documents to generate comprehensive attribute information, then classification accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The inductive classifier extracts only the most important attributes from training documents based on mutual information scores, rather than analyzing all attributes equally. This extraction of essential features maintains classification accuracy while significantly reducing processing time and computational resources.
Solution Approach 2:
The system changes the parameter of attribute selection by using mutual information scoring to rank and select attributes. This parameter change allows the system to focus computational resources on the most discriminative attributes, improving efficiency without sacrificing accuracy.
Data Source
AI summary
A classification system includes a signature-based duplicate detector and an inductive classifier that share attribute information. To perform the duplicate detection and the classification, the duplicate detector and inductive classifier are first initialized by generating a lexicon of attributes for the duplicate detector and a classification model for the classifier. To develop a classification model, a training set of documents of known class are used by the classifier to determine the attributes of the documents that are most useful in classifying an unknown document. The model is developed from these attributes. Attribute information containing the attributes determined by the classifier is then passed to the duplicate detector and the duplicate detector uses the attribute information to generate the lexicon of attributes.