Late-Binding Schema Data Intake System for Flexible Machine Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of diverse machine data generated from various sources, such as system logs, network packets, and sensors, is time-consuming and challenging due to the vast amounts and varied formats of data, which existing tools often address by pre-processing and discarding non-essential data, limiting flexibility and insight.
Innovation Solution
An event-based data intake and query system with a late-binding schema that collects, indexes, and stores machine data as events, allowing for flexible schema definition and extraction rules application at search time, enabling field-searchability and efficient retrieval of minimally processed data across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-processing and discarding non-essential data is performed to reduce data volume, then data analysis efficiency is improved, but data flexibility and insight potential are lost
Solution Approach 1:
The system performs preliminary indexing and metadata extraction on raw data without discarding the original content. Event data is ingested, parsed into structured fields, and indexed for efficient retrieval, while the complete raw data is preserved in its original form. This allows both efficient querying through indexes and full data access when needed, resolving the contradiction between analysis efficiency and data flexibility.
2Adaptability or versatility
If all raw data is stored for later analysis, then data flexibility and insight potential are improved, but data storage and search complexity increase
Solution Approach 1:
The system segments data storage into multiple layers: raw event data is stored in its original form, while simultaneously extracted fields and metadata are stored in separate indexed structures. This segmentation allows the system to maintain complete data flexibility while reducing search complexity by enabling targeted queries against specific indexed fields rather than scanning all raw data.
Solution Approach 2:
The system introduces an intermediary indexing layer between raw data storage and data retrieval operations. Event data is parsed into structured fields with schemas that define data types and relationships. This intermediary structure mediates between the flexibility of storing all raw data and the complexity of searching it, providing efficient query paths while preserving full data access capability.
3Speed
If pre-defined data sets are used for analysis, then analysis speed is improved, but the ability to investigate new data aspects is limited
Solution Approach 1:
The system implements dynamic schema evolution that allows data structures to adapt as new event types and fields are discovered. The schema is not fixed in advance but evolves based on the actual data being ingested, enabling both fast analysis of known data patterns and flexible investigation of new data aspects. This dynamic approach resolves the contradiction between analysis speed and investigation flexibility.
Data Source
AI summary
Techniques are described for enabling an application to automatically generate text narratives explaining risk scores assigned to risk objects. The application uses natural language generation (NLG) techniques to enable the automatic create text narratives providing context and explanation for risk scores. The described approaches use data from a variety of data sources (e.g., risk event indexes, correlation search data, attack framework data, etc.) to create compelling and useful explanations of the risk analysis associated with identified risk objects. These automatically generated text narratives can be readily presented in any number of different interfaces without the need for complex visualizations or user effort to derive the same information. The automatically created text narratives enable users to better understand the risk analysis for particular risk objects, obtain storylines detailing risk objects' activity patterns over time, and to better analyze, triage, and mitigate IT environment risks based on such information.