Interactive Expandable Histogram Timeline for Security Event Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data generated from diverse sources poses challenges due to its vastness and complexity, as existing tools typically pre-process and discard data, limiting flexibility and insight derivation.
Innovation Solution
An event-based data intake and query system with a late-binding schema that collects, indexes, and stores machine data as events, allowing flexible schema definition and extraction rules application at search time, enabling field-searchability and efficient retrieval of minimally processed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is pre-processed and only specified data items are extracted for storage, then retrieval efficiency is improved, but data flexibility and analysis capability are reduced
Solution Approach 1:
The system performs preliminary indexing of all raw data fields during data intake, creating a searchable schema before any analysis is needed. This allows the system to maintain raw data in its original form while enabling efficient field-level searches without requiring pre-processing or data extraction, thus resolving the contradiction between retrieval efficiency and analysis flexibility
Solution Approach 2:
The system employs a dynamic schema that can adapt to different analysis needs at query time. The late-binding schema allows fields to be defined and searched based on actual analysis requirements rather than predetermined structures, enabling the system to efficiently retrieve and analyze diverse data types without sacrificing flexibility
2Adaptability or versatility
If massive quantities of raw data are stored for later analysis, then data flexibility and insight derivation are improved, but search and analysis performance deteriorate
Solution Approach 1:
The system segments raw data into discrete events with structured fields during intake, organizing unstructured data into searchable components. This segmentation allows the system to store comprehensive raw data while enabling efficient queries by breaking down large data volumes into manageable, indexed units that can be quickly searched and retrieved
Solution Approach 2:
The system introduces an intermediary indexing layer that sits between raw data storage and query processing. This indexing mechanism creates a simplified representation of raw data that accelerates search operations without requiring the actual data to be pre-processed or transformed, thus maintaining both flexibility and performance
3Loss of information
If all generated data is retained instead of discarding minimally processed data, then insight derivation capability is improved, but data management complexity increases
Solution Approach 1:
The system implements self-service data management through automated schema inference and field extraction during data intake. The system automatically analyzes incoming data structures, infers appropriate field definitions, and creates indexes without manual configuration, thereby retaining complete raw data while minimizing management complexity through automation
Data Source
AI summary
A display is created and manipulated in connection with an event, such as a notable security event investigation. A plurality of activity indicators for the event are displayed in a time order using both horizontal and vertical directions. Each activity indicator corresponds to a an activity related to the event. A linear display of shapes is provided, each shape corresponding to one or more of the activity indicators, of the same type, in a time sequence. In response to a user action, the time period corresponding to the plurality of shapes is changed so that shapes corresponding to a new time period selected by the user are displayed. Also in response to this user action, only the activity indicators corresponding to the new time period selected by the user are displayed.