Event-Based Data Intake System for Flexible Machine Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of analyzing and searching massive quantities of machine data generated by diverse components in IT environments is complex due to the vast amount of data, varying formats, and the need for efficient retrieval and analysis.
Innovation Solution
An event-based data intake and query system processes, indexes, and stores machine data as events, enabling flexible schema application at search time, field-searchability, and the use of late-binding schema to extract values for specific fields during query execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of raw machine data are stored for later retrieval and analysis, then flexibility in analyzing all generated data is improved, but the complexity of searching and analyzing the data performantly deteriorates
Solution Approach 1:
The patent applies preliminary action by extracting and storing metadata (such as device identifiers, data types, timestamps, and other descriptive information) from raw machine data at the time of data generation. This pre-processing creates an organized framework that enables efficient later retrieval and analysis without requiring complex full-data searches, thus resolving the contradiction between data flexibility and search complexity.
Solution Approach 2:
The patent extracts key metadata elements from the raw machine data and stores them separately in a structured format. This extraction process separates the essential indexing information from the bulk raw data, allowing analysts to quickly search and filter using the extracted metadata while maintaining access to all original generated data when needed.
2Productivity
If specified data items are extracted and stored to facilitate efficient retrieval, then data retrieval efficiency is improved, but the ability to analyze all generated data deteriorates
Solution Approach 1:
The system performs preliminary extraction of metadata at data generation time, creating an efficient search framework that does not discard raw data. This allows both efficient retrieval through metadata filtering and comprehensive analysis when needed, as the extracted metadata serves as an index rather than a replacement for the full data set.
Solution Approach 2:
The patent segments data storage into two components: extracted metadata stored in a structured format for efficient searching, and the complete raw machine data retained for comprehensive analysis. This segmentation enables both fast retrieval operations and full-data analysis capabilities to coexist without mutual interference.
3Quantity of substance
If storage capacity is increased to store all raw data, then data availability for analysis is improved, but the cost and resource requirements deteriorate
Solution Approach 1:
The patent extracts compact metadata from raw machine data and stores it in a structured format that requires minimal storage resources. This extraction allows the system to maintain complete raw data availability while using the small extracted metadata portion for efficient searching and filtering, thereby reducing the effective storage burden without sacrificing data availability.
Solution Approach 2:
The system creates a lightweight copy of essential data characteristics in the form of metadata, which serves as a compact representation enabling efficient data access without requiring duplication or extensive processing of the full raw data sets. This copying approach minimizes storage overhead while maintaining data utility.
Data Source
Figure 1~2
Figure 3A
Figure 3B
AI summary
In various embodiments, a computer-implemented method comprises acquiring, using an edge sensor device, first sensor data associated with a physical device operating within a physical environment, where the edge sensor device includes a first set of sensors of a first sensor type for obtaining the first sensor data, and the edge sensor device is located proximal to the physical device, inputting, by the edge sensor device, the first sensor data into an onboard message bus to publish the first sensor data, wherein a processing device of the edge sensor device maintains the onboard message bus, and upon receipt of the first sensor data, transmitting, by the onboard message bus, the first sensor data onto a network, where the first sensor data is addressed to a first set of one or more subscribers of the onboard message bus, and the one or more subscribers includes a remote server computing system.