Probabilistic Secure Hash Spam Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current email filtering systems using hashing techniques are prone to errors due to user mistakes, spammer evasion methods, and the inability to distinguish between legitimate and spam messages, leading to inaccurate classification and high false positive/negative rates.
Innovation Solution
A system and method that combines probabilistic methods with secure hashes to improve message classification accuracy by using probabilities and thresholds based on complaint rates, focusing on robust hash algorithms that are difficult for spammers to manipulate, such as IP addresses and domain names, and integrating machine learning for enhanced spam detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hashing algorithms are used to filter spam by comparing messages to a database of known spam, then spam detection capability is improved, but false positive rate increases because good messages may accidentally match spam in the database
Solution Approach 1:
The patent introduces probabilistic scoring as an intermediary mechanism between hash matching and final classification. Instead of directly classifying messages as spam or good based on hash matches, the system assigns probabilistic scores that reflect the likelihood of spam, allowing for more nuanced decision-making and reducing false positives from accidental matches
Solution Approach 2:
The patent changes the parameter from binary match/no-match to probabilistic scoring. By transforming the classification parameter from a discrete state (match/mismatch) to a continuous probability value, the system can better distinguish between legitimate messages that accidentally match and actual spam, improving measurement precision
2Measurement precision
If minimum match counts are required before classifying a message as spam, then false positive rate is reduced, but spam detection accuracy deteriorates because the system cannot distinguish between spammers with high complaint rates and legitimate senders with low complaint rates
Solution Approach 1:
The patent changes the classification parameter from raw match counts to complaint rates. By normalizing the metric to account for the total number of messages sent, the system can distinguish between spammers (high complaint rate) and legitimate senders (low complaint rate) even when absolute match counts are similar, improving spam detection accuracy
Solution Approach 2:
The patent performs preliminary normalization of complaint data by calculating complaint rates before classification. This preliminary action of converting raw counts to rates allows the system to make more accurate classifications by accounting for sender volume differences
3Adaptability or versatility
If spammers use techniques to change message aspects such as menu attacks, misspellings, and encoding, then evasion capability is improved, but hash matching effectiveness deteriorates because modified messages no longer match database entries
Solution Approach 1:
The patent introduces probabilistic scoring as an intermediary that aggregates multiple hash match results. Instead of relying on single exact matches, the system combines evidence from multiple partial matches and hash algorithms, making the classification robust to message modifications while maintaining spam detection effectiveness
Data Source
AI summary
Disclosed are signature-based systems and methods that facilitate spam detection and prevention at least in part by calculating hash values for an incoming message and then determining a probability that the hash values indicate spam. In particular, the signatures generated for each incoming message can be compared to a database of both spam and good signatures. A count of the number of matches can be divided by a denominator value. The denominator value can be an overall volume of messages sent to the system per signature for example. The denominator value can be discounted to account for different treatments and timing of incoming messages. Furthermore, secure hashes can be generated by combining portions of multiple hashing components. A secure hash can be made from a combination of multiple hashing components or multiple combinations thereof. The signature based system can also be integrated with machine learning systems to optimize spam prevention.