Synthetic Anomaly Injection for Data Quality Benchmarking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data quality monitoring systems struggle to efficiently and automatically detect anomalies in large datasets, leading to data scars and shocks that impact ML models and decision-making, and traditional methods like manual inspection and rule-based testing are not scalable or reliable.
Innovation Solution
Implementing unsupervised machine learning (ML) for data quality monitoring that automatically learns thresholds and detects anomalies without manual setup, scaling across vast datasets and reducing alert fatigue.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual inspection and rule-based testing are used for data quality monitoring, then detection reliability may be maintained, but scalability and productivity deteriorate
Solution Approach 1:
The patent replaces manual inspection and rule-based testing (mechanical systems) with unsupervised machine learning algorithms. The ML system automatically learns data patterns and detects anomalies without manual rule configuration, enabling scalable automated monitoring while maintaining detection reliability through adaptive learning from data distributions.
Solution Approach 2:
The unsupervised ML system performs self-service by automatically learning thresholds and detecting anomalies without requiring manual setup or continuous rule maintenance. The system autonomously adapts to data patterns and identifies quality issues, eliminating the need for manual intervention while scaling to large datasets.
2Productivity
If unsupervised machine learning is implemented for automated anomaly detection, then productivity and scalability improve, but measurement precision and reliability may worsen due to false positives and negatives
Solution Approach 1:
The patent implements feedback mechanisms where the unsupervised ML system continuously monitors detection performance and adjusts its anomaly detection thresholds based on identified patterns. The system learns from detected anomalies and refines its measurement precision over time, reducing false positives and negatives while maintaining high automated detection efficiency.
3Measurement precision
If traditional rule-based testing is used, then measurement precision can be maintained, but device complexity and ease of operation worsen due to manual setup requirements
Solution Approach 1:
The unsupervised ML system performs self-service by automatically learning data distributions and anomaly thresholds without requiring manual rule configuration. This eliminates the complexity of setting up and maintaining rule-based systems while achieving comparable or superior measurement precision through adaptive learning.
4Ease of operation
If manual data quality monitoring methods are used, then ease of operation can be maintained for small datasets, but productivity and speed deteriorate when scaling to large datasets
Solution Approach 1:
The patent replaces manual monitoring operations with automated unsupervised ML algorithms that process large datasets at high speed. The system maintains ease of operation through automated execution while achieving significant productivity improvements by scaling to enterprise-level data volumes without manual intervention.
Data Source
AI summary
In a general aspect, injecting synthetic anomalies into data is described. In some embodiments, a system identifies a base data set stored in a remote database system and a set of synthetic anomaly injection operations to apply to portions of the base data set. The system submits a set of instructions that are configured to cause the remote database system to perform operations including: creating one or more entries in a lookup table and rerouting received queries attempting to access the one or more portions of the base data set to instead access modified base data while the one or more entries are active. The system causes the remote database system to inactivate the one or more entries in the lookup table such that the remote database system ceases rerouting received queries.