Data Intake System Late-Binding Schema for Raw Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the ability to efficiently search and analyze large sets of raw machine data from diverse sources, as they often pre-process data, discarding unprocessed data and limiting analysis flexibility, and do not provide user-friendly interfaces for visual identification of data subsets.
Innovation Solution
A data intake and query system that processes and stores raw machine data in a flexible schema, allowing for late-binding schema application during search time, enabling field-searchable events and efficient query execution through containerized indexing and search nodes, and utilizing a pipelined search language for real-time data analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data systems pre-process data based on anticipated analysis needs, then data retrieval efficiency is improved, but data analysis flexibility deteriorates because only pre-specified data items are retained
Solution Approach 1:
The system performs preliminary actions by creating indexes and metadata structures during data ingestion, but retains the full raw data in its original form. This allows efficient query processing through pre-built indexes while maintaining the ability to analyze any aspect of the original data later, resolving the contradiction between retrieval efficiency and analysis flexibility
Solution Approach 2:
The system changes the state of data from processed/filtered to raw/unprocessed form, storing complete event data with all fields preserved. This parameter change enables analysts to apply different analysis perspectives and queries to the same data without being constrained by pre-processing decisions, while still benefiting from efficient storage and retrieval mechanisms
2Adaptability or versatility
If massive quantities of raw data are stored for later retrieval, then data analysis flexibility is improved, but storage requirements increase
Solution Approach 1:
The system segments data storage into multiple components: raw event data, extracted fields, indexes, and metadata. Each segment serves a specific purpose and can be optimized independently. This segmentation allows the system to maintain flexible access to all raw data while using efficient storage formats and compression techniques for different data types, reducing overall storage requirements
Solution Approach 2:
The system creates optimized copies of data in different formats and levels of processing. Raw data is preserved in its original form for flexibility, while parallel copies are created with extracted fields and indexes for efficient retrieval. This copying strategy enables the system to meet both flexibility and storage efficiency requirements by using appropriate data representations for different access patterns
3Adaptability or versatility
If tools search data systems separately and collect results over a network, then data from diverse sources can be analyzed, but analysis efficiency deteriorates due to piecemeal collection
Solution Approach 1:
The system merges multiple data sources and search operations into a unified data model and query interface. By consolidating data from diverse sources into a common structure with standardized schemas and metadata, the system enables single-query access to distributed data, eliminating the inefficiency of separate searches and piecemeal result collection while maintaining the ability to analyze data from any source
Data Source
AI summary
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses one or more containerized search nodes to execute the query and stores the results in a data store for combination with additional query results.