Anomaly Detection System Using 2D Statistical Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Detecting and correcting data quality anomalies in large-scale data sets is challenging due to the difficulty in assessing data integrity and identifying material changes, which can negatively impact downstream decision-making.
Innovation Solution
A method and system that involves retrieving data from a database, calculating statistical values over time periods, creating a 2D array of these values, and using machine learning algorithms to detect anomalies, with the ability to correct errors by storing corrected values in a separate database to maintain data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional manual methods are used to assess data quality, then detection accuracy may be maintained, but scalability and productivity deteriorate significantly when dealing with billions of rows across thousands of tables
Solution Approach 1:
The patent segments the massive data quality assessment task into smaller manageable units by dividing data into partitions and processing them in batches. The system breaks down the complex anomaly detection problem into statistical calculations on segmented data subsets, enabling parallel processing across multiple compute nodes while maintaining overall system manageability
Solution Approach 2:
The patent replaces manual mechanical data quality assessment with automated machine learning algorithms. The system uses unsupervised learning models to automatically detect anomalies, substituting human expert analysis with computational algorithms that can process billions of rows across thousands of tables without proportional increases in human resources
2Measurement precision
If comprehensive data quality checks are performed across all data elements, then detection precision improves, but processing time and computational resources increase significantly
Solution Approach 1:
The patent applies partial action by focusing computational resources on detecting anomalies rather than performing exhaustive validation of every data element. The system calculates statistical metrics and identifies deviations from normal patterns, achieving effective anomaly detection without the need to process and validate every single data point in detail
Solution Approach 2:
The patent uses lightweight statistical calculations and approximate algorithms that can be quickly computed and discarded. The system performs rapid statistical measurements (means, variances, percentiles) that are computationally inexpensive compared to exhaustive data validation, enabling fast processing while maintaining adequate detection precision
3Speed
If data is processed in real-time without batching, then responsiveness improves, but system resource consumption and complexity increase
Solution Approach 1:
The patent implements periodic batch processing where data is collected over time intervals and processed in scheduled batches. This approach provides regular updates on data quality status without requiring continuous real-time processing, balancing responsiveness with resource management through time-based periodic execution of detection tasks
Data Source
AI summary
A system and method for detecting anomalies in very large datasets is disclosed. The method includes calculating statistics for data elements in a data set over a range of time periods. These statistics are arranged into a 2D array and analyzed using a machine learning algorithm to detect anomalous regions. The method also includes steps of analyzing time series of the data based on detected anomalous regions, correcting any errors in the datasets, and storing the corrected values in a separate database to maintain data integrity.