Automatic Data Validation Rule Generation via Statistical Outlier Profiling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current enterprise resource planning (ERP) systems require significant manual effort to create validation rules for data validation, which is inefficient in detecting and correcting bad data due to the lack of automated outlier detection and rule generation.
Innovation Solution
The system automatically generates validation rules by profiling data patterns, using statistical tests like Grubbs', Tietjen-Moore, and Generalized Extreme Studentized Deviate tests to identify outliers and create rules that prevent incorrect data entry, and can also generate corrective rules to modify bad data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual validation rules are created for data validation, then data quality can be maintained, but significant manual effort and time are required
Solution Approach 1:
The system performs self-service by automatically generating validation rules through statistical profiling of existing data. The profiling engine autonomously analyzes data patterns, identifies outliers using statistical tests (Grubbs', Tietjen-Moore, GESD), and creates validation rules without human intervention, eliminating the need for manual rule creation while maintaining data quality
Solution Approach 2:
The system performs preliminary action by proactively profiling data and generating validation rules before they are manually created. The automatic rule generation happens in advance by analyzing historical data patterns and preemptively establishing validation criteria, thereby preventing future data quality issues without requiring manual effort
2Measurement precision
If automated outlier detection is implemented, then data accuracy improves, but system complexity increases
Solution Approach 1:
The profiling engine serves as an intermediary between raw data and validation rules. It mediates the complex statistical analysis process by incorporating multiple statistical tests (Grubbs', Tietjen-Moore, GESD) and data profiling capabilities, translating raw data into automated validation rules without requiring end users to understand the underlying complexity
Solution Approach 2:
The system replaces manual mechanical processes of rule creation with automated computational processes. Statistical algorithms and machine learning techniques substitute for human analysts, automatically detecting outliers and generating validation rules through computational analysis rather than manual inspection and rule formulation
Data Source
AI summary
In an example embodiment, a method of automatically generating data validation rules from data stored in a column of a table is provided. Outliers for the data are determined by analyzing a profiling statistic for the data, the profiling statistic having a type. Then it is determined if a predefined limit is exceeded, based on a quantity of the outliers determined for the data through the analysis of the profiling statistic. A data validation rule is then automatically generated based on non-outliers detected in the data through the analysis of the profiling statistic, the generated data validation rule also being based on the type of the profiling statistic. The data validation rule can then be applied to data subsequently entered for the column, causing at least a portion of the data subsequently entered for the column to be rejected.