Anomaly Detection Using Unsupervised Clustering and Tree Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in accurately and efficiently detecting anomalies in transaction data, particularly due to the difficulty in manual identification and the limitations of relying solely on supervised or unsupervised machine learning models, which can be resource-intensive and lack explainability.
Innovation Solution
A combined machine learning system utilizing both unsupervised clustering and supervised single tree classification models to identify anomalies in transaction data, such as income anomalies, without the need for manual labeling, generating executable rules for anomaly detection and verification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised machine learning models are used for anomaly detection, then detection accuracy can be improved through trained models, but resource intensity increases and manual labeling is required
Solution Approach 1:
The system segments the anomaly detection process into two distinct phases: an unsupervised pre-processing phase that clusters data and identifies potential anomalies without resource-intensive manual labeling, and a supervised verification phase that uses trained models only for final classification. This segmentation allows the system to achieve high detection accuracy while reducing overall resource intensity by applying supervised models only where necessary.
Solution Approach 2:
The system performs preliminary unsupervised clustering and anomaly identification before applying supervised machine learning models. By pre-processing the data to group similar transactions and flag potential anomalies in advance, the system reduces the computational resources needed for subsequent supervised training and classification, thereby lowering overall resource intensity while maintaining detection accuracy.
2Use of energy by moving object
If unsupervised machine learning models are used for anomaly detection, then resource intensity is reduced, but verification and explanation capability is lost
Solution Approach 1:
The system implements a feedback mechanism where unsupervised clustering results are fed into supervised verification models, and the outcomes are then fed back for manual review and model retraining. This feedback loop enables the system to maintain low resource intensity through automated unsupervised processing while recovering verification and explanation capabilities through supervised validation and human-in-the-loop verification.
Solution Approach 2:
The system introduces an intermediary verification layer that bridges unsupervised clustering results and final anomaly detection decisions. This intermediary layer, which can be manually verified or validated by domain experts, preserves explanation capability by allowing reviewers to understand and verify the reasoning behind anomaly flags, while still leveraging the resource efficiency of automated unsupervised processing.
3Loss of information
If manual identification of anomalies is performed, then verification and explanation is possible, but productivity decreases and scalability is limited
Solution Approach 1:
The system merges automated unsupervised clustering with manual verification processes to create a hybrid anomaly detection system. By combining the high throughput of automated data processing with the verification capability of human reviewers, the system achieves both high productivity and maintainable explanation quality. The automated system handles the bulk of processing while human experts focus only on verification of flagged anomalies.
4Quantity of substance
If large amounts of transaction data are processed, then detection coverage increases, but manual analysis becomes unfeasible and accuracy decreases
Solution Approach 1:
The system replaces manual mechanical analysis of large datasets with automated unsupervised clustering algorithms that can process vast quantities of transaction data at scale. By substituting human manual review with automated machine learning-based clustering and anomaly identification, the system maintains high detection accuracy even as data volume increases, enabling comprehensive coverage of large datasets without the diminishing returns associated with manual analysis.
Data Source
AI summary
There is provided methods, systems and techniques for optimized anomaly prediction using machine learning. A data set is obtained which corresponds to a query for anomaly detection. Feature classification is performed along with anomaly labelling using an unsupervised clustering technique based on determining similar groups of data and behaviours and determining a distribution for a particular feature of interest in each cluster such as to apply a threshold to each cluster to extract the anomaly data and label same. Once the labelled dataset is generated, a tree classification model is trained based on the labelled data set for detecting anomaly. Once trained, a set of computing model rules may be extracted from the tree classification model to generate a rules executable for anomaly spotting to define combinations of feature characteristics resulting in outlier data so that the rules executable may be applied to new data.