Data Intake Query System for Machine Data Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources is challenging due to the vast amount of data generated by various components in IT environments, such as system logs, network packet data, and sensor data, which requires efficient data processing and storage solutions to facilitate insights and performance monitoring.
Innovation Solution
A data intake and query system that uses a flexible schema to process and store machine data as events with timestamps, allowing for field-searchable and minimally processed data storage, enabling late-binding schema extraction rules to be applied during search time, and utilizing a pipelined search language for efficient query processing across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If data from diverse sources is stored and analyzed to identify correlations, then insights into IT environments are improved, but the complexity of managing and processing the data increases
Solution Approach 1:
The patent segments data from diverse sources into standardized events with common fields (timestamp, host, source, etc.), allowing complex multi-source data to be divided into manageable, uniform units that can be processed independently and then correlated through the common event structure
Solution Approach 2:
The patent introduces an intermediary layer (the standardized event format and common fields) that mediates between diverse data sources and the analysis system, enabling correlation identification without requiring direct integration of each source's complex data structure
2Productivity
If massive quantities of machine data are processed and stored, then operational intelligence is improved, but the resources required for data processing and storage increase
Solution Approach 1:
The patent extracts only the essential and commonly needed fields from massive quantities of machine data (timestamp, host, source, event type, etc.), storing this condensed representation while maintaining the ability to retrieve and analyze the data, thus reducing storage requirements while preserving analytical value
Solution Approach 2:
The patent changes the parameter representation of data by transforming raw machine data into a standardized event format with discrete, searchable fields, enabling efficient indexing and retrieval operations that scale with data volume
3Adaptability or versatility
If data is stored in diverse formats from different sources, then data source versatility is improved, but the difficulty of data retrieval and analysis increases
Solution Approach 1:
The patent creates a universal event format with common fields that can represent data from multiple diverse sources (system logs, network data, sensor data, etc.), allowing a single retrieval mechanism to access and analyze data from any source through the same standardized interface
Data Source
AI summary
Systems and methods are disclosed for implementing a data stream correlation user interface. The data stream correlation user interface enables users to view information from two sets of records, and identify fields in the two sets of records that can be matched together to “glue” together multiple records. For example, a user may specify that values in an “AcctID” field in one set of records can be matched to values in an “Account_ID” field of a second set of records. Additional identifying fields may be selected, such that multiple values can be chained together. The system can match the records of multiple sets together using designated fields, enabling users to view how many records from one set have a corresponding record in another set.