Abnormality Detection Engine for Network Message Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for detecting unwanted computer message traffic, such as spam, are inefficient and costly, as they rely on content scanning, blacklists that are easily evaded, and are ineffective with shorter messages or false positives.
Innovation Solution
A system and method utilizing an Abnormality Detection Engine (ADE) with components like fanout, fanin, error response, bandwidth variation, and message content detectors to identify abnormal message traffic patterns, providing a cost-effective and efficient means to detect and report sources of unwanted messages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If filtration software scans every message for content and checks with databases, then message identification accuracy is improved, but computer time consumption increases
Solution Approach 1:
The detection system is divided into multiple specialized detectors (fanout detector, fanin detector, error response detector, bandwidth variation detector, message content detector) that each analyze specific aspects of message traffic. This segmentation allows parallel processing of different detection tasks, improving overall efficiency while maintaining comprehensive analysis accuracy.
Solution Approach 2:
The system performs preliminary analysis by examining traffic patterns, fanout/fanin relationships, and bandwidth variations before conducting detailed content scanning. This preliminary detection filters out obvious abnormal messages early in the process, reducing the number of messages that require resource-intensive content verification.
2Productivity
If Bayesian or heuristic filters are used to statistically identify unwanted messages, then filtering efficiency is improved, but detection accuracy for shorter messages deteriorates
Solution Approach 1:
The system combines multiple detection approaches including statistical analysis (Bayesian/heuristic filters), pattern analysis (fanout/fanin detectors), and content-based detection. By merging these different detection methods, the system maintains high filtering efficiency while improving accuracy for short messages through the complementary strengths of multiple detectors working together.
3Object-generated harmful factors
If blacklists of IP addresses are used to block unwanted messages, then blocking effectiveness is improved, but reliability deteriorates due to address reuse
Solution Approach 1:
The system transitions from static blacklist filtering to dynamic pattern-based detection that continuously monitors and adapts to changing traffic patterns. Detectors track fanout/fanin relationships, error responses, and bandwidth variations over time, allowing the system to identify spammers based on their behavioral patterns rather than fixed addresses, thus maintaining reliability when IP addresses are reassigned.
4Measurement precision
If content scanning is performed on every message, then detection accuracy is improved, but network resource consumption increases
Solution Approach 1:
The system applies partial content scanning only to messages that exhibit suspicious patterns detected by other detectors. The fanout/fanin detectors, error response detectors, and bandwidth variation detectors filter out clearly normal or clearly abnormal messages without full content analysis, performing content scanning only on borderline cases where additional verification is needed.
Data Source
AI summary
The present invention relates generally to a system and method for the monitoring of email and other message traffic on a network. The intent of the monitoring to determine if message traffic is abnormal, thus indicating unwanted messages such as spam. A number of methods may be utilized by the invention to recognize unwanted messages, including the calculation of fanout, the number of messages sent by a unique host, unique email address or domain. Also included is fanin, the number of messages received from unique hosts, unique domains or unique email addresses. Further components consider the number of error messages received from a host, variations in bandwidth from a host, and variations in message content from a host.