Spam Filtering Using Insignificant Shingles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing spam filtering methods are inadequate in distinguishing between legitimate and spam messages, leading to false detections and inefficiencies, as they often misclassify mass mailing services and fail to effectively counter evolving spam tactics.
Innovation Solution
The system identifies and removes insignificant text portions and shingles from messages, using databases of non-spam patterns and shingles to generate a reduced set for filtering, employing methods like Bayes and cluster filtering to determine spam content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If black lists are used for spam filtering, then filtering completeness is improved, but false detection rate increases
Solution Approach 1:
The patent extracts and removes insignificant text portions (headers, signatures, footers, watermarks) from messages before filtering. This extraction principle resolves the contradiction by eliminating common elements that appear in both spam and legitimate messages, thereby reducing false detections while maintaining filtering effectiveness on the remaining significant content.
Solution Approach 2:
The patent segments the message into significant and insignificant portions, applying different processing approaches to each. By dividing the message structure and selectively filtering only the significant parts, the system achieves both high filtering completeness and low false detection rates.
2Productivity
If message clustering is used for spam detection, then detection speed is improved, but accuracy decreases
Solution Approach 1:
The patent removes insignificant text portions before clustering analysis. This extraction ensures that clustering is performed only on significant message content, improving detection accuracy while maintaining the speed benefits of clustering by working with a reduced, more relevant dataset.
3Reliability
If content filtering is used to analyze all message parts, then filtering thoroughness is improved, but processing time increases
Solution Approach 1:
The patent extracts and removes insignificant text portions (headers, signatures, footers, watermarks) before applying content filtering. This preprocessing step reduces the volume of text requiring thorough analysis, thereby maintaining filtering thoroughness on significant content while significantly reducing overall processing time.
Solution Approach 2:
The patent segments the message into significant and insignificant portions, applying thorough content filtering only to the significant parts. This selective filtering approach maintains thoroughness where needed while minimizing processing time by excluding irrelevant sections.
Data Source
AI summary
Disclosed system and methods for detecting spam using shingles. An example system identifies in a received message one or more insignificant text portions based on a text pattern database storing defined insignificant text patterns not containing spam; removes at least a portion of the one or more identified insignificant text portions from the message to generate an abridged and canonized message; generates a set of shingles from the abridged and canonized message; identifies in the set of shingles one or more shingles based on a shingles database storing defined insignificant shingles that occur only in messages not containing spam; removes one or more identified shingles from the set of shingles to generate a reduced set of shingles upon detecting the one or more identified shingles matching at least one of the defined insignificant shingles; and determines whether the received message contains spam based on the reduced set of shingles.