Data Anomaly Detection Using Random Forest and FP-Growth

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for identifying data anomalies in datasets are inefficient, prone to errors, and inconsistent, making it difficult to accurately predict correct or missing data and overall data quality due to cumbersome manual identification of relationships and frequent false positives.

Innovation Solution

Utilizing associative machine learning algorithms like Random Forest and Frequent Pattern Growth to automatically identify relationships between data points, generate predictive indicators, and flag anomalies based on frequency analysis.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual identification of data relationships is used, then data anomalies can be identified, but the process is cumbersome and time-consuming

Engineering Contradiction:

Improvedata anomaly identification accuracyVSAvoidtime to identify data relationships

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces manual mechanical identification of data relationships with automated machine learning algorithms. The system uses ML models to automatically discover relationships between data points, validate records, and identify anomalies without human intervention, thereby resolving the contradiction between identification accuracy and time consumption.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system enables self-service by allowing the data validation process to perform automatically without manual guidance. The machine learning models autonomously learn data relationships, apply validation rules, and generate anomaly reports, eliminating the need for manual relationship identification while maintaining high accuracy.

Inventive Principle:

Principle #25Self-service

2Reliability

If manual identification of data relationships is used, then data anomalies can be identified, but the process is inefficient and yields errors

Engineering Contradiction:

Improvedata anomaly identification consistencyVSAvoiddata validation throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent replaces inefficient manual processes with automated machine learning-based validation systems. These systems consistently apply learned relationships across large datasets, eliminating human errors and inconsistencies while processing data at machine speed, thereby simultaneously improving reliability and productivity.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system incorporates feedback mechanisms where machine learning models continuously learn from validated data patterns and anomaly corrections. This feedback loop improves the accuracy and consistency of relationship identification over time, while the automated nature maintains high processing throughput without manual intervention.

Inventive Principle:

Principle #23Feedback

3Productivity

If automated machine learning algorithms are used, then data relationships can be identified efficiently, but system complexity increases

Engineering Contradiction:

Improvedata validation speedVSAvoidsystem architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the complex data validation task into distinct modular components: data preprocessing modules, multiple specialized machine learning models for different relationship types, validation rule engines, and anomaly reporting systems. This segmentation allows each component to be independently optimized and managed, reducing overall system complexity while maintaining high processing speed.

Inventive Principle:

Principle #1Segmentation

4Reliability

If comprehensive data validation is performed, then data quality can be improved, but computational resources are consumed

Engineering Contradiction:

Improvedata quality prediction accuracyVSAvoidcomputational processing power

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent applies partial validation by focusing computational resources on identifying and validating the most critical data relationships and high-risk anomaly patterns. The machine learning models prioritize validation of key business rules and frequently occurring data patterns, achieving high data quality improvement while consuming fewer computational resources than exhaustive validation would require.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS12450526B2Systems and methods for predicting correct or missing data and data anomalies

Publication Date: 2025.10.21 COLLIBRA BELGIUM BV

AI summary

The present disclosure is directed to systems and methods for predicting and correcting data anomalies. In one example aspect, data is received by the system. The system may analyze the data by profiling the data for certain profiling statistics (e.g., min, max, mean, cardinality, etc.). At least one machine-learning algorithm (e.g., a Random-Forest algorithm) may be applied to the profiled data to identify potential relationships among certain data columns in the data. Once certain relationships are identified, the data that is related may be extracted to form an itemset. A second machine-learning algorithm (e.g., Frequent Pattern Growth algorithm) may be applied to the itemset to identify certain frequencies of related values in the itemset. Low frequency values may indicate anomalies in the dataset. If an anomaly is detected, the system may be configured to provide an intelligent remedial action, such as substituting certain values and/or filling in a missing value.