Dependency Graph Models for Machine Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data generated from diverse sources in IT environments is challenging due to the vast amount of data types and formats, leading to inefficiencies in data processing and retrieval.
Innovation Solution
A computerized method for generating a dependency graph using ingested data, employing machine learning techniques to predict future metrics, and utilizing a late-binding schema for flexible data analysis and search capabilities across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If massive quantities of raw machine data are stored for later retrieval and analysis, then data analysis flexibility and completeness are improved, but data storage requirements and system complexity increase
Solution Approach 1:
The patent segments the massive raw machine data into discrete events with specific fields and types. Each event is structured with standardized fields (timestamp, host, source, etc.) making the data manageable and analyzable while preserving flexibility. This segmentation allows the system to handle large volumes of data without proportionally increasing complexity.
Solution Approach 2:
The patent introduces an intermediary layer (the structured event model and metadata schema) between the raw data storage and the analysis processes. This intermediary standardizes the data representation, enabling flexible analysis without directly increasing the complexity of the storage system itself.
2Productivity
If data is pre-processed and extracted based on anticipated analysis needs, then data retrieval and analysis efficiency are improved, but data completeness and flexibility are reduced
Solution Approach 1:
The patent implements a dynamic approach where the level and type of data processing can be adjusted based on the specific analysis needs. The system can perform minimal processing during ingestion and more extensive processing during analysis, or vice versa, allowing optimization for different scenarios without sacrificing flexibility.
Solution Approach 2:
The patent changes the parameter of data processing intensity from a fixed pre-processing approach to a variable approach. The system can adjust processing depth based on the analysis requirements, enabling efficient retrieval when needed while maintaining the option for comprehensive analysis when flexibility is prioritized.
3Adaptability or versatility
If raw data is minimally processed with segmentation and metadata annotation, then data storage efficiency and retrieval flexibility are improved, but data processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions during data ingestion by segmenting data into events and adding essential metadata (timestamp, host, source type). This preliminary structuring enables efficient retrieval and analysis later without requiring extensive processing at query time, thus reducing the processing time burden when flexibility is needed.
4Measurement precision
If accurate source type assignment is performed during data ingestion, then extraction rule accuracy and search result precision are improved, but data processing complexity and time increase
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically determines source types and assigns appropriate extraction rules without requiring manual configuration for each data source. The system uses metadata and pattern recognition to self-configure, improving accuracy while reducing the operational complexity burden.
Data Source
AI summary
Machine data is collected from multiple sources of an operating environment such as an information technology system, factory floor, or the like, into a data intake and query system, in one embodiment. Metrics representative of the environment are included in or derived from the data. Users may interact with an interface to depict a representation of various metrics and interdependencies and that depiction is reflected in a computer storage model. Changes to the computer storage model based on the user interaction may also result in training of a machine learning model according to the user interaction, the machine learning model configured to determine a prediction, classification or clustering of a result of a first search query by utilizing a result of at least a second search query as input to the machine learning model.