Aggregate Notable Events for Streamlined Security Playbook Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data generated by modern computing environments is challenging due to the vast amount of data types and formats, which are often discarded during pre-processing, limiting the ability to investigate different aspects of the data.
Innovation Solution
A data intake and query system utilizing a late-binding schema that stores minimally processed machine data and applies extraction rules during search time, enabling flexible analysis and the use of a common information model across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-specified data items are extracted and stored during pre-processing, then data retrieval efficiency is improved, but data flexibility and analysis capability are worsened
Solution Approach 1:
The system performs preliminary indexing of all machine data during pre-processing, creating a searchable structure that enables both efficient retrieval and flexible analysis. The late-binding schema is prepared in advance but applied dynamically during search time based on actual query needs.
Solution Approach 2:
The schema binding is delayed until search time rather than being fixed during pre-processing. This dynamic approach allows the system to adapt the data structure and extraction rules based on the specific analysis needs of each query, resolving the contradiction between efficiency and flexibility.
2Quantity of substance
If massive quantities of machine data are stored for later analysis, then data completeness and analysis flexibility are improved, but system complexity and processing challenges are worsened
Solution Approach 1:
The system segments machine data into discrete events with standardized fields during pre-processing. This segmentation organizes the massive data volume into manageable units that can be efficiently stored, retrieved, and processed on demand without overwhelming system complexity.
Solution Approach 2:
The patent introduces an intermediary indexing layer that sits between the raw machine data and the analysis queries. This intermediary structure handles the complexity of processing massive data by providing a standardized access interface, while the actual data remains in its original format for complete retention.
3Quantity of substance
If pre-processing discards portions of machine data, then storage requirements are reduced, but data investigation capability is worsened
Solution Approach 1:
The system extracts only the essential structural information (schema, field types, relationships) during pre-processing while retaining the complete machine data in its original form. This extraction creates a lightweight indexing structure that enables flexible investigation without requiring extensive storage for redundant data copies.
Data Source
AI summary
Techniques are described for an IT and security operations application to automatically generate aggregate (or “bulk,”“group,” or “composite”) notable events by identifying notable events sharing common characteristics and aggregating the related notable events into a single aggregate notable event entity that can be displayed and operated upon. The IT and security operations application identifies related notable events based on notable events generated by a common correlation search, notable events having common event attributes, based on user-specified relatedness criteria, or other such criteria. Once identified, in some embodiments, the IT and security operations application displays, in notable event lists and other interfaces, a singular aggregate notable event to users representing each of the identified related notable events.