AI Anomaly Detection With Rule-Based Data Reconfiguration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in managing data quality due to anomalies and noise in datasets, which can lead to biased and inaccurate AI model training, resulting in flawed analytics and operational risks.
Innovation Solution
A data management platform using AI models to identify, evaluate, and correct anomalies by comparing observed data patterns with reference patterns, generating reconfiguration commands to align data with expected norms, and detecting out-of-distribution data to improve data quality and model reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If AI models are trained on large amounts of data, then model performance improves, but data quality deteriorates due to presence of anomalies and noise
Solution Approach 1:
The system performs preliminary data cleaning and anomaly detection before training AI models. By identifying and removing anomalies, missing values, and other data quality issues in advance, the system ensures that only high-quality data is used for model training, thus maintaining both large data volume and high data quality
Solution Approach 2:
The system introduces an intermediary data quality assessment layer between data collection and model training. This intermediary layer uses AI-based anomaly detection and data cleaning mechanisms to filter and prepare data, ensuring that only clean data progresses to the training stage
2Measurement precision
If manual data cleaning is performed to improve data quality, then anomaly detection accuracy improves, but processing time increases
Solution Approach 1:
The system enables self-service data cleaning through automated AI-based anomaly detection and correction mechanisms. The system automatically identifies, evaluates, and corrects anomalies without requiring manual intervention, thus maintaining high detection accuracy while significantly reducing processing time
Solution Approach 2:
The system replaces manual data cleaning processes with automated AI-based mechanisms. Instead of human analysts manually reviewing and cleaning data, the system uses machine learning models and algorithms to automatically detect and correct anomalies, achieving both speed and accuracy
3Productivity
If data cleaning processes are automated to reduce processing time, then productivity improves, but control over data quality deteriorates
Solution Approach 1:
The system implements feedback mechanisms where AI models continuously learn from cleaned data and improve their anomaly detection capabilities. The system provides feedback loops that allow it to refine its cleaning processes over time, maintaining high data quality control while operating at automated speed
Solution Approach 2:
The system dynamically adjusts cleaning parameters and thresholds based on the characteristics of the data being processed. By adapting parameters to specific datasets and contexts, the system maintains optimal data quality control across different scenarios while preserving automated processing speed
Data Source
AI summary
The systems and methods disclosed herein receive a dataset including an observed set of values for a set of variables. The system can use a first set of AI models to identify a set of anomalies in the observed set of values by comparing an observed set of patterns against multiple reference patterns. The system can use a second set of AI models to evaluate the identified anomalies by comparing an observed set of association rules with an expected set of association rules. The system can use a third set of AI models to generate reconfiguration commands to remove the identified anomalies. The reconfiguration commands can be automatically executed to modify the observed association rules to align with the expected association rules.