Enterprise Threat Detection Pattern Creation via Log Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise threat detection systems face challenges in efficiently analyzing large volumes of log data to identify suspicious behavior due to the need for complex filtering and pattern creation, which can be resource-intensive and time-consuming, especially when dealing with raw data.
Innovation Solution
A method that allows for interactive free text searching of log data, enabling the creation of enterprise threat detection patterns by selecting relevant search results, adding event filters based on normalized log data, and setting a relative time range, thereby facilitating efficient pattern completion and alert generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complex filtering and pattern creation are performed on raw data, then detection accuracy is improved, but processing time and resource consumption increase
Solution Approach 1:
The system performs preliminary normalization of log data into a standardized format before pattern creation. By preparing the data in advance and converting it to normalized events, the system reduces the computational burden during actual threat detection, thereby decreasing processing time while maintaining detection accuracy through the use of normalized data structures.
Solution Approach 2:
Normalized log data serves as an intermediary between raw logs and detection patterns. The normalization layer acts as a mediator that transforms diverse raw log formats into a unified representation, enabling efficient pattern matching without requiring complex filtering operations on the original raw data, thus reducing processing resources and time.
2Reliability
If complex filtering and pattern creation are performed on raw data, then detection accuracy is improved, but resource consumption increases
Solution Approach 1:
The system performs preliminary normalization of log data into a standardized format before pattern creation. By preparing the data in advance and converting it to normalized events, the system reduces the computational burden during actual threat detection, thereby decreasing processing time while maintaining detection accuracy through the use of normalized data structures.
Solution Approach 2:
Normalized log data serves as an intermediary between raw logs and detection patterns. The normalization layer acts as a mediator that transforms diverse raw log formats into a unified representation, enabling efficient pattern matching without requiring complex filtering operations on the original raw data, thus reducing processing resources and time.
3Adaptability or versatility
If free text search is performed on raw logs, then search flexibility is improved, but processing complexity increases
Solution Approach 1:
The search process is segmented into two distinct phases: first, flexible free-text searching is performed on raw logs to identify relevant events; second, the identified events are converted to normalized events for pattern creation. This segmentation allows the system to maintain search flexibility on raw data while separating the complexity of normalization to a dedicated processing stage, reducing overall processing complexity.
Solution Approach 2:
Normalized log data serves as an intermediary between raw logs and detection patterns. The normalization layer acts as a mediator that transforms diverse raw log formats into a unified representation, enabling efficient pattern matching without requiring complex filtering operations on the original raw data, thus reducing processing resources and time.
Data Source
AI summary
Search results are received from an initiated free text search of log data from one or more logs, where the free text is performed using search terms entered into a free text search graphical user interface. A set of at least one search result is selected from the search results containing an event desired to be identified in a completed enterprise threat detection (ETD) pattern. A forensic lab application is rendered to complete an ETD pattern. An event filter is added for an event type based on normalized log data to a path. A relative ETD pattern time range is set and an ETD pattern is completed based on the added event filter.