Multi-level log analysis for software anomaly detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current detection systems for unauthorized access and transmission of sensitive data are inadequate, often relying on crude keyword searches or manual log reviews, which fail to identify unauthorized activity in a timely manner, leading to potential reputational, monetary, and legal risks for organizations.
Innovation Solution
A computer-implemented method and system that analyzes logs from multiple communication abstraction levels (application, session, and network) using multiple anomaly classifiers, with an ensemble model to identify anomalies and alert human reviewers, enhancing the accuracy and speed of detecting sensitive data exposure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If crude keyword searches or manual log reviews are used for detection, then the system complexity is low, but the detection precision and speed are insufficient
Solution Approach 1:
The system segments log analysis into multiple communication abstraction levels (application, session, network layers), with each level processed by specialized anomaly classifiers. This segmentation allows complex detection to be broken down into manageable components, improving detection precision while organizing system complexity in a structured manner.
Solution Approach 2:
The system transitions from single-level keyword searching to multi-level log analysis across different communication abstraction dimensions. By analyzing logs at application, session, and network layers simultaneously, the system achieves comprehensive anomaly detection that crude single-dimensional methods cannot accomplish.
2Productivity
If multi-level log analysis with ensemble models is implemented, then the detection speed and accuracy improve, but the computational resources and processing time increase
Solution Approach 1:
The system performs preliminary filtering and anomaly classification at each communication layer before combining results in the ensemble model. By pre-processing logs at application, session, and network levels separately, the system reduces the computational burden on the final ensemble decision, improving overall detection speed while managing resource consumption.
Solution Approach 2:
The system applies anomaly classifiers selectively at different log levels rather than analyzing all logs uniformly. By focusing computational resources on identifying and analyzing only anomalous patterns at each layer, the system achieves high detection accuracy without processing every log entry at maximum depth, thus balancing productivity with resource usage.
3Reliability
If automated anomaly identification is implemented, then the detection reliability improves, but the ease of operation decreases due to system complexity
Solution Approach 1:
The system implements automated anomaly identification that operates independently without requiring manual intervention at each detection stage. The ensemble model automatically processes multi-level logs, identifies anomalies, and generates alerts, making the detection process self-service and highly reliable while reducing the operational burden on human operators.
Solution Approach 2:
The system incorporates feedback mechanisms where detection results and anomaly patterns are continuously analyzed to refine the ensemble model's performance. This feedback loop improves detection reliability over time by learning from past anomalies, while the automated nature of the feedback process maintains ease of operation without requiring manual tuning.
Data Source
AI summary
A computer-implemented method for identifying a use anomaly potentially exposing sensitive data is disclosed. The method comprises receiving data comprising logs of a communication involving a computing device, where the logs comprise distinct logs of at least three communication abstraction levels. At least three anomaly classifiers are operated for logs from each of the at least three communication abstraction levels. An ensemble model is used to identify an anomaly in the communication, by processing output from each of the at least three anomaly classifiers. The various logs from a moment in time when the anomaly occurred are collated, and a graphical user interface is generated for reviewing the identified anomaly and collated logs. A human reviewer is then alerted that an anomaly has been identified.