Spam Filtering Using Shingles and Text Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current spam filtering methods are inadequate in distinguishing between legitimate and spam messages, often resulting in false positives and inefficiencies due to the constant evolution of spamming techniques, leading to network overload and fraud.
Innovation Solution
A system and method utilizing shingles, where insignificant text portions such as signatures and auto-signatures are removed from messages to generate a reduced set of shingles, which are then filtered using a database of non-spam characteristics to determine if a message contains spam, enhancing filtering accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If black lists are used to filter spam messages, then filtering completeness is improved, but false positive rate increases causing legitimate messages to be blocked
Solution Approach 1:
The message is segmented into multiple components (subject line, body text, signatures, headers) and processed differently. Insignificant portions like signatures are removed separately from significant content, allowing the filter to focus on discriminatory features while ignoring common legitimate message elements that would cause false positives in blacklisting approaches.
Solution Approach 2:
Insignificant text portions such as signatures and auto-signatures are extracted and removed from the message before analysis. This extraction eliminates noise that would otherwise interfere with accurate spam detection, allowing the filter to concentrate on meaningful content that actually distinguishes spam from legitimate messages.
2Productivity
If message clustering is used to detect identical mass messages, then spam detection speed is improved, but false positive rate increases as legitimate services are misidentified
Solution Approach 1:
Different parts of the message are treated with different levels of analysis. Significant content portions undergo detailed shingle-based analysis while insignificant portions are removed. This local differentiation allows fast clustering on key discriminative features while avoiding misclassification based on common elements like signatures that appear in both spam and legitimate messages.
Solution Approach 2:
Common elements such as signatures and auto-signatures are extracted and removed before clustering analysis. This prevents these ubiquitous elements from causing legitimate services to be misidentified as spam, while still enabling fast detection of actual spam patterns in the remaining significant content.
3Measurement precision
If complete message analysis is performed for spam filtering, then detection accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
Insignificant text portions are extracted and removed before generating shingles and performing filtering analysis. This reduction in data volume directly decreases processing time and computational resources required, while the selected significant portions maintain sufficient information for accurate spam detection.
Solution Approach 2:
The message processing is segmented into distinct stages: removal of insignificant portions, generation of shingles from remaining text, and filtering based on shingle analysis. This segmentation allows efficient processing by focusing computational effort only on significant message portions that contribute to accurate classification.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are a system and methods for detecting spam using shingles. In one aspect, the system receives an electronic message including at least a text portion. The system identifies in the received message insignificant text portions. The system then removes identified insignificant text portions to generate an abridged message. The system then generates a set of shingles from the abridged message. The system then identifies in the generated set of shingles one or more shingles that occur only in messages not containing spam. The system then removes one or more identified shingles from the generated set of shingles to generate a reduced set of shingles. The system then performs spam filtering of the reduced set of shingles to determine whether the received message contains spam.