Malicious Event Classification via Time-Decayed Feature Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current SIEM and log management tools face challenges in effectively classifying malicious network events due to the vast amount of log data, requiring manual configuration of rules that are time-consuming and prone to imbalance with network changes, leading to noise and confusion in information security monitoring.
Innovation Solution
Implementing machine learning techniques to mine relevant features from historical and real-time log data from SIEM solutions and event sensors, creating predictive models that classify malicious behavior and provide actionable insights for network monitoring and incident response.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deterministic correlation rules are manually configured to monitor network security, then security monitoring capability is improved, but time consumption and operational complexity increase significantly
Solution Approach 1:
The system automatically generates and configures correlation rules by analyzing historical log data and identifying patterns of malicious activity, eliminating the need for manual rule configuration. The machine learning model autonomously performs the tasks that previously required security analysts to manually create and maintain deterministic correlation rules.
Solution Approach 2:
The system dynamically adjusts rule parameters and thresholds based on learned patterns from historical data, allowing the security monitoring system to adapt to changing threat landscapes without manual intervention. This enables automatic optimization of detection sensitivity and reduces the time required to configure effective rules.
2Measurement precision
If deterministic correlation rules are configured to achieve effective security monitoring, then detection accuracy is improved, but adaptability to network changes deteriorates
Solution Approach 1:
The system transitions from static deterministic rules to dynamic machine learning models that continuously learn from new data. The model adapts its detection criteria based on evolving network patterns and emerging threats, maintaining high detection accuracy while automatically adjusting to network changes without requiring manual rule updates.
Solution Approach 2:
The system pre-trains the machine learning model on historical log data to establish baseline detection capabilities before deployment. This preliminary learning phase enables the system to achieve high detection accuracy from the outset, while the model's inherent ability to continue learning ensures ongoing adaptability to network changes.
3Loss of information
If manual configuration of correlation rules is performed to reduce noise in log data, then signal-to-noise ratio is improved, but operational costs increase
Solution Approach 1:
The machine learning system automatically analyzes log data to identify patterns and generate optimized correlation rules, eliminating the need for expensive manual configuration by security analysts. The system self-adjusts detection parameters to maximize signal-to-noise ratio without requiring operational intervention.
Solution Approach 2:
The system replaces the mechanical process of manual rule configuration with automated machine learning algorithms. This substitution eliminates the need for human analysts to manually tune correlation rules, significantly reducing operational costs while maintaining or improving detection quality.
4Productivity
If the number of log entries increases to capture more network activity, then monitoring coverage is improved, but system complexity and noise increase
Solution Approach 1:
The machine learning model extracts only the most relevant features and patterns from the large volume of log data, separating signal from noise. By focusing computational resources on the most informative aspects of the data, the system maintains high monitoring coverage while reducing the effective complexity of the analysis required.
Solution Approach 2:
The system dynamically adjusts analysis parameters and feature selection based on the characteristics of the incoming log data, optimizing the balance between monitoring coverage and computational complexity. This enables the system to handle increasing volumes of logs efficiently without proportionally increasing system complexity.
Data Source
AI summary
A system for classifying events on a computer network includes an event clustering engine for receiving event and log data related to identifiable actors from a security information and event management (SIEM) or log management module and selecting behavioral groupings of the event and log data. An affinity-based feature generation module assigns a value to each identifiable actor based on occurrences within predetermined time intervals of the identifiable actors having the selected behavioral grouping. A time-based weighting decay module applies a time decaying function to the assigned values for each identifiable actor. A feature engineering storage module stores information relating to the identifiable actors and their associated time-decayed values. A machine learning module generates a prediction model based on information received from the event clustering engine and the time-based weighting decay module, and the prediction model is utilized by a prediction engine on a computer to predict and classify received event and log data as malicious or non-malicious.