Email Spam Classification via Cluster Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The proliferation of spam emails burdens both users and providers, as they are time-consuming and costly due to storage and bandwidth usage, with existing filtering methods often ineffective against varied and randomized spam content.
Innovation Solution
A spam filtering method that clusters similar emails and uses a set of test emails to determine a spam proportion threshold, classifying emails based on similarity and misclassification costs, reducing overall system costs by evaluating only a subset of emails and using accurate duplicate detection techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all received emails are individually classified using traditional spam filtering methods, then classification accuracy may be maintained, but system processing time and computational resources increase significantly
Solution Approach 1:
The patent segments the email classification process into two distinct phases: (1) clustering phase where emails are grouped into clusters of similar content using duplicate detection techniques, and (2) classification phase where only representative emails from each cluster are classified. This segmentation reduces the total number of emails requiring individual classification while maintaining accuracy through cluster representation.
Solution Approach 2:
The patent performs preliminary clustering of emails into groups of similar content before the classification step. By organizing emails into clusters based on content similarity, the system prepares the data structure in advance so that subsequent classification only needs to evaluate representative samples from each cluster rather than every individual email.
2Reliability
If traditional spam filtering methods are used to handle varied and randomized spam content, then individual email classification is performed, but the system becomes less effective against bulk spam campaigns
Solution Approach 1:
The patent merges multiple similar emails into single clusters based on content similarity. By combining emails with identical or similar content into clusters, the system treats bulk spam campaigns as unified entities rather than individual messages, thereby improving effectiveness against coordinated spam attacks while adapting to content variations through the clustering algorithm.
3Loss of information
If every received email is stored and individually processed, then complete email archives are maintained, but storage and bandwidth costs increase substantially
Solution Approach 1:
The patent creates cluster representative copies rather than storing and processing every individual email. By identifying and classifying only the representative email from each cluster, the system maintains the essential information content while significantly reducing storage requirements and processing overhead associated with duplicate or near-duplicate emails.
4Productivity
If a small number of test emails are selected from each cluster, then processing speed increases, but the proportion of spam emails may not accurately reflect the entire cluster
Solution Approach 1:
The patent adjusts the number of test emails selected from each cluster based on cluster size and spam prevalence characteristics. By dynamically changing the sampling parameter (number of test emails) according to specific cluster properties, the system optimizes the balance between processing speed and measurement accuracy, ensuring sufficient statistical representation while maintaining high productivity.
Data Source
AI summary
An e-mail filter is used to classify received e-mails so that some of the classes may be filtered, blocked, or marked. The e-mail filter may include a classifier that can classify an e-mail as belonging to a particular class and an e-mail grouper that can detect substantially similar, but possibly not identical, e-mails. The e-mail grouper determines groups of substantially similar e-mails in an incoming e-mail stream. For each group, the classifier determines whether one or more test e-mails from the group belongs to the particular class. The classifier then designates the class to which the other e-mails in the group belong based on the results for the test e-mails.