CNN-Based Anomaly Detection in Fielded Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data cleaning techniques require significant human intervention and are not scalable, making them inefficient for large volumes of data used in business intelligence systems, as they typically apply only one or a limited number of criteria to identify anomalous field values.
Innovation Solution
A convolutional neural network (CNN) is trained to automatically detect and suggest corrections for anomalous field values in a dataset using multiple similarity measures, reducing the need for manual intervention and improving scalability by employing anomaly detection and suggestion networks that utilize similarity measures like semantic, syntactic, and format similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If current data cleaning techniques are used, then data cleaning can be performed, but significant human intervention is required and scalability is poor
Solution Approach 1:
The patent replaces manual mechanical data cleaning processes with an automated machine learning system that uses trained models to detect and correct anomalies. The system substitutes human operators with automated algorithms that can process data at scale without proportional increases in human resources.
Solution Approach 2:
The data cleaning system performs self-service by automatically detecting anomalies and generating corrections without requiring human intervention. The trained machine learning models autonomously identify problematic data points and suggest corrections, enabling the system to clean data independently.
2Measurement precision
If multiple similarity measures are applied, then anomaly detection accuracy improves, but computational complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-training machine learning models on historical data before deployment. The models learn from labeled examples during a training phase, so that when deployed for actual data cleaning, they can quickly apply multiple similarity measures without requiring complex real-time computations for each data point.
3Reliability
If manual data cleaning is performed, then data quality can be improved, but time and effort requirements are significant
Solution Approach 1:
The patent replaces manual data cleaning operations with automated machine learning systems that can process large volumes of data rapidly. The system maintains high data quality standards while reducing the time investment from hours or days of manual work to automated processing that occurs in minutes or seconds.
Data Source
AI summary
The technology disclosed determines which field values in a set of unique field values for a particular field in a fielded dataset are anomalous using six similarity measures. A factor vector is generated per similarity measure and combined to form an input matrix. A convolutional neural network processes the input matrix to generate evaluation vectors. A fully-connected network evaluates the evaluation vectors to generate an anomaly scalar for a particular unique field value. Thresholding is applied to anomaly scalar to determine whether the particular unique field value is anomalous.