Sourcetype Inference for Machine Data Intake
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of diverse machine data generated by various components in IT environments is challenging due to the vast amount of data types and formats, leading to inefficiencies in data retrieval and analysis.
Innovation Solution
A data intake and query system that utilizes a late-binding schema and sourcetype inference to process and store machine data, allowing for flexible schema development and refinement during search time, enabling efficient extraction and analysis of data across disparate sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is pre-processed and stored with predetermined schemas, then data retrieval efficiency is improved, but data analysis flexibility deteriorates
Solution Approach 1:
The patent implements a dynamic schema evolution mechanism where the data schema is not fixed in advance but adapts automatically based on incoming data characteristics. The system performs schema inference from sample data, allowing the schema to evolve and change as new data types and formats are encountered, thereby maintaining both retrieval efficiency and analysis flexibility
Solution Approach 2:
The system changes the schema parameters dynamically by inferring data types, formats, and structures from actual incoming data rather than using predetermined parameters. This allows the schema to adjust its parameters (data types, validation rules, storage formats) based on the actual data being processed, resolving the contradiction between efficiency and flexibility
2Adaptability or versatility
If all raw data is stored for later analysis, then data analysis flexibility is improved, but data processing complexity increases
Solution Approach 1:
The system performs preliminary schema inference and data classification actions during the data ingestion phase rather than during analysis. By inferring schemas upfront and organizing data according to inferred types before storage, the system reduces the complexity of later processing while maintaining the ability to analyze all raw data
Solution Approach 2:
The system enables self-service data organization where the incoming data itself provides the information needed for its own classification and schema definition. The schema inference mechanism automatically determines data types and structures from the data content, eliminating the need for complex manual preprocessing while maintaining flexibility
3Measurement precision
If sourcetype is assigned accurately during data intake, then data processing precision is improved, but data intake time increases
Solution Approach 1:
The system applies partial sourcetype assignment during data intake by inferring sourcetypes from available sample data or data characteristics without requiring complete analysis of all data attributes. This partial action provides sufficient accuracy for initial processing while minimizing the time added during intake, with the capability to refine assignments later
Data Source
AI summary
As described herein, a portion of machine data of a message may be analyzed to infer, using an inference model, a sourcetype of the message. The portion of machine data may be generated by one or more components in an information technology environment. Based on the inference, a set of extraction rules associated with the sourcetype may be selected. Each extraction rule may define criteria for identifying a sub-portion of text from the portion of machine data of the message to produce a value. The set of extraction rules may be applied to the portion of machine data of the message to produce a result set that indicates a number of values identified using the set of extraction rules. Based on the result set, at least one action may be performed on one or more of inference data associated with the inference model and one or more messages.