Interactive Chart Creation from Field-Searchable Data Events
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources is challenging due to varying data types and formats, making it time-consuming and inefficient to derive insights from large volumes of machine data generated by IT environments.
Innovation Solution
An event-based data intake and query system with a late-binding schema that processes and indexes machine data, allowing flexible schema development and extraction of insights at search time, enabling field-searchable events and a common information model across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of raw machine data are stored for later retrieval and analysis, then data flexibility and analytical depth are improved, but data retrieval and search efficiency deteriorate due to varying data types and formats
Solution Approach 1:
The patent segments machine data into discrete events with standardized schemas. Each event is parsed and organized into structured fields (e.g., host, source, sourcetype, timestamp) that can be independently indexed and searched. This segmentation transforms unstructured raw data into manageable, searchable units that maintain flexibility while enabling efficient retrieval through field-based queries.
Solution Approach 2:
The patent applies parameter changes by transforming diverse data formats into a standardized event structure with consistent field types. Data is converted from varying formats into uniform parameters (host string, source string, sourcetype string, timestamp numeric) that can be efficiently indexed and searched. This parameter standardization enables flexible data storage while maintaining high retrieval efficiency through structured field searches.
2Productivity
If pre-processing is applied to reduce data volume before storage, then storage costs and retrieval time are reduced, but data flexibility and available insights deteriorate due to discarded information
Solution Approach 1:
The patent extracts essential information from raw machine data into standardized event fields while preserving the complete original data. The extraction process identifies and pulls out key parameters (host, source, sourcetype, timestamp, message) into structured fields for efficient indexing, while the full raw data is retained in the original event. This allows flexible analysis of extracted fields without losing access to the complete unprocessed data for future insights.
Solution Approach 2:
The patent performs preliminary parsing and schema application during data ingestion to prepare data for efficient future retrieval. Events are pre-processed to extract and index key fields (host, source, sourcetype, timestamp) while maintaining the complete raw data. This preliminary action enables fast field-based searches and flexible querying without requiring re-processing of raw data, balancing processing efficiency with data flexibility.
3Adaptability or versatility
If diverse data types from multiple sources are ingested without standardization, then data completeness and flexibility are improved, but search and analysis complexity increases
Solution Approach 1:
The patent creates a universal event schema that can accommodate diverse data types from multiple sources through a common structure. The standardized event format with fields like host, source, sourcetype, timestamp, and message serves as a multi-functional container that can represent various machine data types (logs, metrics, traces) from different sources. This universality enables consistent searching and analysis across diverse data sources without increasing complexity, as all data follows the same queryable structure.
Data Source
AI summary
A system generates a user interface that enables a user to generate a chart from one or more statements of a data processing package. Via one or more user interactions with the user interface, the system may receive one or more chart parameters for a chart. Using a statement from the data processing package and the one or more chart parameters, the system may generate an additional statement and append the generated statement to the data processing package to form an enriched data processing package. The system may communicate the enriched data processing package to a search service for execution. The system may display the results in an interactive chart.