Edge ML Classifier for Network Scanning Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network anomaly detection systems face challenges in distinguishing between legitimate and illegitimate scanning activities, often resulting in false positives due to the impracticality of manual whitelisting and the dynamic nature of network behaviors.
Innovation Solution
Implementing edge-based machine learning by training a classifier using labeled traffic data to differentiate between legitimate and illegitimate scanning activities, which is then deployed across the network to reduce false positives and eliminate the need for manual whitelisting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual whitelisting is used to identify legitimate scanning activity, then legitimate scans can be distinguished from illegitimate ones, but the system becomes cumbersome to manage and cannot adapt to dynamic network behaviors
Solution Approach 1:
The system performs self-service by automatically learning and updating the classifier model using network traffic data without requiring manual intervention. The classifier trains itself to distinguish legitimate from illegitimate scanning activity, eliminating the need for manual whitelist maintenance while adapting dynamically to new scanning patterns.
Solution Approach 2:
The classifier model is dynamically updated by retraining it periodically with recent network traffic data, allowing the system to adapt to changing network behaviors and new legitimate scanning patterns without manual intervention. This dynamic adaptation resolves the contradiction by maintaining high accuracy while eliminating manual management overhead.
2Speed
If anomaly detection is deployed at all network nodes, then scanning activity can be detected in real-time, but the computational resources and complexity increase significantly
Solution Approach 1:
The system segments the anomaly detection functionality by deploying lightweight classifier models at network nodes rather than implementing full anomaly detection systems. Each node receives a pre-trained classifier that requires minimal computational resources, enabling real-time detection without significant increases in device complexity or resource requirements.
3Measurement precision
If the classifier model is frequently updated to adapt to new scanning patterns, then detection accuracy improves, but the time and resources required for training increase
Solution Approach 1:
The classifier model is retrained periodically at scheduled intervals rather than continuously or on every new scanning pattern detected. This periodic retraining approach maintains detection accuracy by updating the model with recent traffic data while limiting the time and computational resources spent on training, resolving the contradiction between accuracy and training overhead.
Data Source
AI summary
In one embodiment, a device in a network receives an indication that a network anomaly detected by an anomaly detector of a first node in the network is associated with scanning activity in the network. The device receives labeled traffic data associated with the detected anomaly that identifies whether the traffic data is associated with legitimate or illegitimate scanning activity. The device trains a machine learning-based classifier using the labeled traffic data to distinguish between legitimate and illegitimate scanning activity in the network. The device deploys the trained classifier to the first node, to distinguish between legitimate and illegitimate scanning activity in the network.