Adaptive Thresholding for Real-Time KPI Outlier Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools lack the capability to efficiently search and analyze large sets of raw machine data for insights, as they often require pre-processing that discards significant amounts of data and do not provide intuitive, visual interfaces for identifying data subsets of interest.
Innovation Solution
A data intake and query system that utilizes a flexible schema to store and index raw machine data, allowing for real-time search and analysis through a pipelined search language, enabling users to extract insights from all generated data without pre-defined formats, and providing a user-friendly interface for querying and visualizing results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is pre-processed to extract specified data items for efficient retrieval, then retrieval efficiency is improved, but data completeness deteriorates as most generated data is discarded
Solution Approach 1:
The system performs preliminary indexing of all raw machine data without discarding any information. Indexes are created in advance that enable efficient retrieval while preserving the complete raw data set for later analysis, thus achieving both retrieval efficiency and data completeness
Solution Approach 2:
The system creates indexed copies of raw machine data that facilitate efficient searching and retrieval. These indexes are supplemental structures that do not replace the original raw data but provide fast access paths while the complete raw data remains stored for comprehensive analysis
2Measurement precision
If tools are designed for specific data analysis needs, then analysis precision is improved, but adaptability deteriorates as they cannot handle diverse data types
Solution Approach 1:
The system implements a universal data processing platform that can handle diverse data types including machine data, logs, metrics, and unstructured data. The unified storage and indexing approach allows the same system to perform multiple analysis functions across different data types without requiring specialized tools for each data category
Solution Approach 2:
The system uses configurable parameters and metadata schemas that can be adjusted to handle different data types and analysis requirements. By changing parameters such as data format specifications, indexing strategies, and query configurations, the system adapts to diverse data types while maintaining consistent processing precision
3Adaptability or versatility
If massive quantities of raw data are stored for later retrieval, then data flexibility is improved, but storage cost increases
Solution Approach 1:
The system segments raw machine data into manageable units such as time-based buckets or event types, organizing them in a hierarchical structure. This segmentation enables efficient storage management where data is divided into smaller chunks that can be stored, retrieved, and processed more effectively, reducing the overhead of managing massive data sets while preserving flexibility
Data Source
AI summary
Systems and methods are described for performing adaptive thresholding on key performance indicator (KPI) values using an online machine learning algorithm as the KPI values or the data from which the KPI values are derived is being ingested. For example, the system can identify outliers in a moving window of KPI values. To implement the adaptive thresholding, the system may identify seasonality and/or trend components in historical KPI values. When a new KPI value is obtained, the system may remove the identified seasonality and/or trend components from the KPI value, and determine whether the modified KPI value is an outlier using sketches or quantiles. The system can then repeat this process for each subsequently received KPI value.