Server Log Analysis for Automated Threat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security threat detection methods are inefficient in identifying advanced malicious codes and behavior-based threats, requiring manual log analysis and being unable to effectively respond to diverse and rapidly changing internal network environments, leading to delayed detection and increased damage from security incidents.
Innovation Solution
A machine learning-based approach that preprocesses server system logs, stores them in a Hadoop distributed file system, extracts feature values, trains a machine learning model, and calculates risks using observation levels to automatically detect abnormal behaviors and generate warnings, enabling timely and optimized responses to security threats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual log analysis is performed by operators, then security threats can be detected, but the detection process requires considerable time and cannot respond rapidly to security incidents
Solution Approach 1:
The patent replaces the mechanical manual analysis process with an automated machine learning system. The machine learning model automatically analyzes server system logs to detect security threats, substituting human operators with an automated computational system that processes logs rapidly without sacrificing detection accuracy.
Solution Approach 2:
The system enables self-service by allowing the machine learning model to autonomously perform security threat detection without requiring human intervention. The model automatically processes logs, identifies anomalies, and generates detection results, making the security monitoring process self-sufficient and eliminating the time-consuming manual analysis step.
2Quantity of substance
If traditional log collection methods are used, then log information can be gathered, but the system cannot effectively respond to diverse and rapidly changing internal network environments
Solution Approach 1:
The patent implements dynamics by making the machine learning model adaptable to changing network environments. The model can be retrained with new log data from varying internal network environments, allowing it to dynamically adjust its detection capabilities to respond effectively to diverse and rapidly changing conditions without requiring system redesign.
Solution Approach 2:
The system utilizes parameter changes by adjusting the machine learning model's parameters and retraining it with log data from different network environments. This allows the model to adapt its detection parameters to match the characteristics of varying network environments, maintaining effectiveness across diverse conditions.
3Productivity
If automated monitoring of large amounts of events is implemented, then detection speed improves, but the false detection rate increases
Solution Approach 1:
The patent applies feedback by using the machine learning model to continuously learn from detection results and adjust its parameters accordingly. The model receives feedback from both positive detections and false alarms, allowing it to refine its detection criteria over time. This feedback mechanism enables the system to maintain high detection speed while progressively reducing the false detection rate through iterative improvement.
4Measurement precision
If machine learning models are trained with leveled baseline values, then abnormal behavior detection accuracy improves, but the model training complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-processing the training data to create leveled baseline values before model training. This preliminary preparation of the data, including normalization and baseline establishment, simplifies the actual model training process while ensuring high detection accuracy. The complex data preparation work is done upfront, making the subsequent training less complex.
Data Source
AI summary
The present disclosure relates to a method of performing machine learning-based observation level measurement including: a log preprocessing step; a log file linkage step of processing a log file to store the log file in a HDFS, and linking the processed log file to a big data storage; a feature value extraction step of requesting an inquiry of a raw log, and extracting a feature value for a normal behavior from the inquired raw log; a model training step of normalizing the extracted feature value to level a baseline value for the normal behavior, and training a machine learning model based on the leveled baseline value; and a risk calculation step of determining, when a log that violates the leveled baseline value is detected, that an abnormal behavior is detected so as to calculate a risk for the detected abnormal behavior.