Dictionary DGA Detection Pipeline With Filtering and Cached Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to efficiently and cost-effectively detect dictionary DGA domain names due to their resemblance to legitimate domain names, leading to increased latency and computational costs in network security systems.
Innovation Solution
A machine learning-based detection system that utilizes a trained model and preprocessing filters to classify domain names, incorporating pDNS data and NLP techniques, caches predictions, and dynamically scales resources to reduce latency and cost.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a trained machine learning model is used to detect dictionary DGA domain names, then detection accuracy is improved, but computational cost and latency increase
Solution Approach 1:
The detection system is segmented into multiple components: a preprocessing filter that performs initial screening using pDNS data and NLP techniques, and a trained machine learning model that processes only the filtered subset of domain names. This segmentation allows the system to maintain high detection accuracy while reducing the computational burden and latency associated with running the full model on all domain names.
Solution Approach 2:
The preprocessing filter performs preliminary analysis of domain names before they reach the trained machine learning model. By applying filtering criteria based on pDNS data and NLP techniques in advance, the system eliminates obviously benign or malicious domain names early in the detection pipeline, reducing the number of domain names that require computationally expensive model processing and thereby reducing overall latency.
2Measurement precision
If a trained machine learning model is used to detect dictionary DGA domain names, then detection accuracy is improved, but computational cost increases
Solution Approach 1:
The detection system is segmented into multiple components: a preprocessing filter that performs initial screening using pDNS data and NLP techniques, and a trained machine learning model that processes only the filtered subset of domain names. This segmentation allows the system to maintain high detection accuracy while reducing the computational burden and latency associated with running the full model on all domain names.
Solution Approach 2:
Instead of applying the full computational power of the trained machine learning model to all incoming domain names, the system applies partial action by using a lightweight preprocessing filter to handle the majority of domain names. The expensive model is invoked only for a subset of domain names that pass through the filter, representing a partial application of computational resources that is sufficient to maintain detection accuracy while significantly reducing overall computational cost.
3Reliability
If all domain names are processed through the trained model, then detection thoroughness is improved, but processing speed decreases
Solution Approach 1:
The preprocessing filter performs preliminary analysis of domain names before they reach the trained machine learning model. By applying filtering criteria based on pDNS data and NLP techniques in advance, the system eliminates obviously benign or malicious domain names early in the detection pipeline, reducing the number of domain names that require computationally expensive model processing and thereby reducing overall latency.
Solution Approach 2:
The preprocessing filter acts as a self-service mechanism that automatically screens and categorizes domain names before they reach the main detection model. By using readily available pDNS data and NLP techniques, the filter independently handles routine classification tasks, allowing the trained model to focus on more complex cases and improving overall processing throughput.
Data Source
AI summary
A service includes a trained model comprising a classifier that predicts whether domain names are dictionary DGA generated. Using passive DNS data and/or a heuristic analysis based on natural language processing of the domain name, the service filters domain names that are not candidate (i.e., potential) dictionary DGA domain names out of the detection pipeline. There domain names are thus classified without being fed into the model for more computationally expensive processing. Domain names that are not filtered out are queued for input into an instance of the model and classification by the model, with the queued domain names processed in small batches and load balanced across model instances. Predicted domain name classes output by the model are cached for subsequent cache reads to avoid multiple runs of the model for one domain name.