Volumetric Clustering for Large-Scale DNS Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing analysis pipelines are unable to efficiently process large-scale real-world DNS data to identify hidden relationships between domains and detect new malicious domains.
Innovation Solution
A method for volumetric clustering of large-scale DNS data using a trained K-shape model, which aggregates and filters domains based on hit patterns to identify actionable malicious clusters and detect unknown malicious domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing analysis pipelines are used to process DNS data, then the processing can be performed with simple methods, but the pipelines are unable to efficiently process large-scale data to identify hidden relationships between domains
Solution Approach 1:
The analysis pipeline is divided into distinct processing stages: data collection from DNS servers, hit pattern extraction, aggregation by domain, filtering to remove duplicates, and clustering using K-shape algorithm. Each stage processes data independently and passes results to the next stage, enabling efficient handling of large-scale DNS data while maintaining manageable complexity through modular design
Solution Approach 2:
The patent transforms DNS data from traditional tabular format into time-series hit patterns, adding a temporal dimension to the analysis. By representing domain queries as sequences of hits over time and applying volumetric clustering in this extended dimensionality space, the system can identify hidden relationships and malicious clusters that would be invisible in conventional analysis
2Measurement precision
If all domains are analyzed in detail to detect malicious domains, then detection accuracy improves, but the computational resources and time required increase significantly
Solution Approach 1:
The system extracts only the essential hit pattern features from DNS query data - specifically the timing and frequency of queries to each domain. By focusing analysis on these extracted temporal patterns rather than examining all domain attributes in detail, the pipeline achieves accurate malicious domain detection while maintaining efficient processing speed
Solution Approach 2:
The patent applies filtering operations that remove duplicate entries and focus computational resources on suspicious patterns. The K-shape clustering algorithm performs partial analysis by identifying clusters of domains with similar hit patterns, allowing the system to detect malicious groups without exhaustively analyzing every single domain in detail
Data Source
AI summary
In an approach, a processor receives a set of normal domains, a set of suspicious domains, and a set of malicious domains; labels each domain of the set of normal domains as normal producing a labelled set of normal domains and each domain of the set of suspicious domains and the set of malicious domains as malicious producing a labelled set of malicious domains; samples a preset percentage of the labelled set of normal domains producing a sampled set of normal domains; aggregates the sampled set of normal domains and the labelled set of malicious domains producing a set of aggregated domains; filters the set of aggregated domains using hit size, inter-arrival-time, and univariate volumetric filters producing a set of filtered domains; and determines a cluster of a set of clusters to which each of the set of filtered domains is to be assigned using a trained K-shape model.