Precalculated Metrics for Anomaly Detection in Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing large dynamic datasets is time-consuming and resource-intensive, particularly when detecting anomalies, as existing methods require iterative querying and row-by-row data scans, leading to inefficiencies in computing resources and time.
Innovation Solution
A data analysis method that precalculates and stores summary metrics, allowing for anomaly detection based on user-defined criteria applied to these metrics rather than raw data, reducing the need for row-by-row scans and improving efficiency by using precomputed data statistics for anomaly identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If iterative querying and row-by-row data scans are used for anomaly detection, then comprehensive data analysis is achieved, but computing resources and time consumption increase significantly
Solution Approach 1:
The patent precalculates and stores summary metrics (such as min, max, mean, standard deviation) for data sets before anomaly detection is needed. When anomaly detection is required, the system applies anomaly detection criteria to these precomputed metrics rather than scanning the entire data set row-by-row. This preliminary computation significantly reduces the time and computing resources needed for anomaly detection while maintaining detection accuracy.
2Reliability
If row-by-row data scans are performed for anomaly detection, then all data points are examined, but the process becomes tedious and resource-intensive
Solution Approach 1:
The patent extracts and stores summary metrics from the complete data set in advance. These metrics capture essential statistical properties of the data. During anomaly detection, the system only needs to examine these extracted metrics rather than the entire data set, significantly reducing detection time while maintaining reliability through the use of statistically significant summary information.
3Productivity
If summary metrics are precalculated and stored, then anomaly detection speed improves, but additional storage requirements are created
Solution Approach 1:
The patent transforms the data representation from storing all individual data points to storing aggregated statistical parameters (summary metrics) such as minimum, maximum, mean, and standard deviation. This parameter transformation dramatically reduces the storage volume required while enabling efficient anomaly detection through criteria applied to these compact statistical representations.
Data Source
AI summary
In the general, the subject matter of the present disclosure relates to a data analysis tool that helps consumers, such as data scientists and engineers, understand datasets better. The disclosed data analysis framework/tool surfaces metrics to data consumers to visually inspect and understand large datasets more efficiently. In addition to the pre-computed and pre-collected metrics on given datasets or tables, the disclosed data analysis tool may also provide a way to detect various anomalies associated with the datasets.