Malicious Network Traffic Detection via Weak Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The detection of malicious network communication faces challenges due to the scarcity and high cost of labeled datasets, with existing methods relying on obsolete domain-level labeling that is easily evaded by malware, leading to poor performance and frequent updates required to keep pace with evolving threats.
Innovation Solution
A Multiple Instance Learning (MIL) approach using weak annotations from Internet domain blacklists, whitelists, and sandboxing analysis to train a detector that classifies network traffic records into 'bags' based on domain labels, optimizing the decision boundary to minimize false negatives and false positives, and employing a Neyman-Pearson detector to ensure a low false positive rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If domain-level labeling is used to compile blacklists of malicious domains, then the labeling process becomes easier and faster, but the detection accuracy deteriorates because malicious domains change frequently and evasion techniques are easily implemented
Solution Approach 1:
The patent segments the detection task from the labeling task. Domain-level labels are used only to segment and identify potential malicious groups, while the actual detection is performed by a flow-level classifier that analyzes individual network flows using multiple features. This segmentation allows easy labeling at the domain level while maintaining accurate detection at the flow level.
Solution Approach 2:
The patent introduces an intermediary flow-level classifier that bridges the gap between coarse domain-level labels and fine-grained detection requirements. The classifier uses domain labels as one of many features and combines them with flow-level characteristics to make accurate detection decisions, acting as a mediator that translates weak labels into reliable detection.
2Measurement precision
If extensive labeled datasets are collected and manually annotated by security experts, then the training data quality improves, but the time and cost required for data preparation increases significantly
Solution Approach 1:
The patent applies partial labeling by using domain-level labels instead of complete flow-level labels for all samples. This partial action is sufficient because the flow-level classifier can learn from the domain labels combined with flow features, achieving good detection performance without the excessive time investment required for complete manual annotation of every network flow.
Solution Approach 2:
The system uses automated domain blacklist/whitelist services to provide labels without requiring manual security expert analysis for each sample. The domain labels are obtained through self-service mechanisms using publicly available intelligence, significantly reducing the time and human resources needed for data preparation while maintaining sufficient training quality.
3Adaptability or versatility
If domain blacklists are updated frequently to keep pace with evolving malware, then the detection system remains current, but the system complexity and maintenance burden increase
Solution Approach 1:
The patent segments the update burden by separating domain-level updates from flow-level detection. Domain blacklists can be updated independently and frequently without affecting the flow-level classifier structure. This segmentation allows easy adaptation to new threats while maintaining a stable, reusable detection model that reduces overall system maintenance complexity.
Solution Approach 2:
The patent creates a dynamic system where domain labels can be updated frequently to reflect evolving threats, while the flow-level classifier remains relatively stable. The classifier adapts to new domain labels automatically during normal operation, allowing the system to remain current without requiring complex reconfiguration or retraining procedures.
Data Source
AI summary
Techniques are presented that identify malware network communications between a computing device and a server utilizing a detector process. Network traffic records are classified as either malware or legitimate network traffic records and divided into groups of classified network traffic records associated with network communications between the computing device and the server for a predetermined period of time. A group of classified network traffic records is labeled as malicious when at least one of the classified network traffic records in the group is malicious and as legitimate when none of the classified network traffic records in the group is malicious to obtain a labeled group of classified network traffic records. A detector process is trained on individual classified network traffic records in the labeled group of classified network traffic records and network communication between the computing device and the server is identified as malware network communication utilizing the detector process.