Flexible Schema Anomaly Detection for Machine Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing and searching massive quantities of machine data from diverse sources is challenging due to the complexity and volume of data generated by thousands of components in computing environments, requiring efficient automation for anomaly detection and insights extraction.
Innovation Solution
A data intake and query system that uses a flexible schema to process and store machine data as events, enabling late-binding schema for extraction rules application at search time, allowing for field-searchability and efficient anomaly detection across disparate data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a flexible schema with late-binding extraction rules is used, then adaptability to diverse data sources is improved, but device complexity increases
Solution Approach 1:
The system performs preliminary actions by pre-defining extraction rules and schemas that can be automatically applied to incoming data streams. The anomaly detection system pre-processes data by applying extraction rules to identify relevant fields before full analysis, reducing the complexity of real-time processing while maintaining adaptability to diverse data sources through configurable schema templates.
Solution Approach 2:
The patent introduces an intermediary layer consisting of extraction rules and schema mappings that mediate between diverse data sources and the anomaly detection engine. This intermediary layer translates various data formats into a standardized internal representation, allowing the system to handle diverse sources without increasing core system complexity.
2Productivity
If automated anomaly detection is implemented, then productivity in analyzing machine data is improved, but device complexity increases
Solution Approach 1:
The anomaly detection system is segmented into distinct functional modules: data ingestion components, schema processing modules, extraction rule engines, anomaly detection algorithms, and visualization interfaces. This segmentation allows each component to be independently optimized and managed, improving overall productivity while controlling complexity through modular design.
Solution Approach 2:
The system implements universal anomaly detection capabilities that can analyze multiple types of machine data from various sources using the same core detection algorithms. The extraction rules and schema mappings provide multi-functionality, allowing a single system to handle diverse data formats and anomaly types without requiring separate specialized systems for each data source.
3Ease of operation
If late-binding schema is used for extraction rules, then ease of operation is improved, but measurement precision may be affected
Solution Approach 1:
The system performs preliminary validation and type checking of extraction rules before they are applied to data. Schema definitions pre-specify expected data types, formats, and validation criteria, ensuring measurement precision is maintained even though the binding occurs late in the process. This preliminary preparation allows flexible query formulation without sacrificing extraction accuracy.
Data Source
AI summary
A computerized method is disclosed that includes operations of obtaining a data set, selecting candidate parameter pairs to be analyzed, wherein the candidate parameter pairs include a window length and a sensitivity multiplier, and wherein the window length is a number of data points, performing an anomaly detection process for each candidate parameter pair including importing each candidate parameter pair into a predetermined search query thereby generating a set of populated predetermined search queries, wherein the predetermined search query is configured to perform the anomaly detection process, executing each search query of the set of populated predetermined search queries on the data set to obtain a set of anomaly detection results, and scoring each anomaly detection result by applying a set of heuristics to the set of the anomaly detection results, and generating an auto-tuned search query by selecting a first candidate parameter pair based on a score of each of the set of anomaly detection results and importing the first candidate parameter pair into the predetermined search query.