Two-Phase Spam Filtering via Token Frequency Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional network-level spam detectors are ineffective against spammers who use legitimate source addresses, leading to false positives and failures in filtering spam messages, as they rely solely on message source information.
Innovation Solution
A two-phased approach using a modified naïve Bayesian classifier for initial message classification, followed by a token frequency-based analysis to identify spam messages, which reduces false positives by capturing dynamic characteristics of messages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional network-level spam detectors use IP source address and URI filtering, then they can filter messages from known spam sources, but they become ineffective against spammers using legitimate source addresses, leading to false positives
Solution Approach 1:
The spam detection system is divided into two independent phases: Phase 1 uses a statistical classifier (naïve Bayesian) to perform initial message classification based on content, while Phase 2 applies token frequency analysis specifically to messages classified as potential spam. This segmentation allows each phase to specialize in different detection aspects, improving overall accuracy without false positives from source-based filtering
Solution Approach 2:
The patent introduces an intermediary statistical classifier that bridges the gap between traditional source-based filtering and content-based analysis. This classifier uses message content tokens and statistical probabilities to determine spam likelihood, serving as a mediator that doesn't rely on source addresses but still provides reliable filtering through content analysis
2Productivity
If spam filters rely solely on message source information, then they can quickly identify known spammers, but they fail to detect spammers using legitimate addresses and cannot distinguish between bulk message distributors and actual spammers
Solution Approach 1:
The system performs preliminary classification in Phase 1 using a statistical classifier that quickly analyzes message content tokens and assigns spam probabilities. This preliminary action filters out obvious spam cases rapidly while flagging suspicious messages for more detailed analysis in Phase 2, maintaining speed while improving accuracy
Solution Approach 2:
The patent changes the detection parameters from static source-based identifiers (IP addresses, URIs) to dynamic content-based parameters (token frequencies, statistical probabilities). By analyzing the frequency and distribution of tokens within message content, the system can adapt to different spammer behaviors and distinguish between legitimate bulk senders and actual spammers based on message patterns rather than source identity
Data Source
AI summary
Embodiments are directed towards classifying messages as spam using a two phased approach. The first phase employs a statistical classifier to classify messages based on message content. The second phase targets specific message types to capture dynamic characteristics of the messages and identify spam messages using a token frequency based approach. A client component receives messages and sends them to the statistical classifier, which determines a probability that a message belongs to a particular type of class. The statistical classifier further provides other information about a message, including, a token list, and token thresholds. The message class, token list, and thresholds are provided to the second phase where a number of spam tokens in a given message for a given message class are determined. Based on the threshold, the client component then determines whether the message is spam or non-spam.