Spear Phishing Email Detection Using Related Anomaly Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing detection methods for spear phishing in enterprise networks are ineffective due to limited labeled training sets and the challenge of super imbalanced data, leading to high false positive rates and resource-intensive machine learning requirements.
Innovation Solution
A system using Related Anomaly Score (RAS) to identify suspicious emails through feature vectors, prioritizing based on map position, and applying customized spear phishing rules to reduce false positives and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning techniques are used for spear phishing detection, then detection accuracy may improve, but resource consumption and training requirements increase significantly
Solution Approach 1:
The detection system is segmented into two distinct components: a lightweight RAS-based detection module for initial screening and a machine learning-based analysis module for deep inspection of suspicious emails. This segmentation allows the system to apply computationally intensive ML techniques only when necessary, reducing overall resource consumption while maintaining detection accuracy.
Solution Approach 2:
The RAS (Reputation and Anomaly Score) calculation is performed as a preliminary action before ML-based analysis. By pre-calculating reputation scores for senders, domains, and URLs, and computing anomaly scores based on email characteristics, the system filters out obvious legitimate emails early, preventing them from consuming ML processing resources.
2Measurement precision
If machine learning techniques are used for spear phishing detection, then detection capability improves, but false positive rate increases due to imbalanced data
Solution Approach 1:
The system transforms the imbalanced classification problem by changing parameters: instead of directly classifying emails as phishing or legitimate, it calculates RAS values (reputation scores and anomaly scores) that continuously represent email suspiciousness. This parameter transformation allows the system to set flexible thresholds and reduces the impact of class imbalance on false positive rates.
Solution Approach 2:
The RAS calculation serves as an intermediary between raw email features and final phishing classification. By introducing reputation scores (for senders, domains, URLs) and anomaly scores as intermediate metrics, the system creates a buffer that smooths out the effects of imbalanced training data, leading to more reliable detection with fewer false positives.
3Device complexity
If traditional phishing detection methods are used, then system complexity remains low, but detection speed and effectiveness decrease
Solution Approach 1:
The detection pipeline is segmented into sequential stages with increasing complexity: RAS-based screening (low complexity) → ML-based analysis (high complexity). This segmentation enables the system to achieve fast detection for most emails through the simple RAS stage, while reserving complex ML analysis for a small subset of suspicious emails, thus improving overall detection speed without requiring the entire system to be complex.
Solution Approach 2:
The system dynamically adjusts its detection approach based on email characteristics. Emails are first evaluated using the lightweight RAS method, and only those exceeding certain anomaly thresholds are subjected to more complex ML analysis. This dynamic, adaptive approach optimizes detection speed by avoiding unnecessary complex analysis of legitimate emails while ensuring thorough inspection of suspicious ones.
Data Source
AI summary
Emails suspected to include a spear phishing attack are identified from the stream of incoming emails using a Related Anomaly Score (RAS). The RAS is calculated by identifying feature vectors from the stream of incoming emails associated with a sender of the email and a link of the email. The suspicious spear phishing emails are mapped by feature vectors and prioritizing according to map position. For reliability, in one case, relative distances are calculated between suspicious emails, and if a relative distance between the specific email and prioritized suspicious emails exceeds a predetermined distance threshold, take a security action based on spear phishing rules on the filtered highest suspicious emails, and if the relative distance does not exceed the predetermined distance threshold, take a second security action including forwarding the specific email for standard phishing analysis.